I cannot speak to what the law is in all jurisdictions, but I can answer this question as it applies in the jurisdiction where I practice divorce and family law (Utah).
Custody battles can arise between married and unmarried parents. Without conducting deep analysis of the question of whether one can you file a defamation case against the other parent while one is involved in a child custody case, I am aware of no legal prohibition against it and while it’s rarely advisable to engage in two different lawsuits with the same person at the same time, I can think of no reason why the pending child custody case (just a child custody case, not a divorce case) would cause any fatal legal or procedural impediments to successfully prosecuting a defamation case is pending simultaneously.
In the divorce case of Noble v. Noble, 761 P.2d 1369 (Utah 1988). Here is the pertinent part of that decision from the Utah Supreme Court:
Glen’s first argument is that Judge Tibbs improperly considered Elaine’s tort claims in the divorce action. We held in Walther v. Walther, 709 P.2d 387, 388 (Utah 1985), that it is improper to try a tort claim, as such, within a divorce action. Accord Lord v. Shaw, 665 P.2d 1288, 1291 (Utah 1983). Tort claims, which are legal in nature, should be kept separate from divorce actions, which are equitable in nature. As a practical matter, if spouses have tort claims pending against each other which are likely to have a bearing on the outcome of the divorce action, those claims should be resolved prior to the divorce proceedings.
In this case, Glen contends that Judge Tibbs combined the two proceedings and used the property division and alimony award as a means of giving Elaine damages properly attributable to her tort claims. The record does not support this contention. Judge Tibbs was fully informed that the tort claims were being tried in a separate action before Judge Ballif. For that reason, Judge Tibbs stated in his findings and conclusions that he had avoided consideration of the merits of the tort claims qua tort claims, and our review of the record provides us with no cause to dispute that assertion. It is true that some of the facts relevant to the tort claims were considered in the divorce proceeding, including Elaine’s medical and living expenses incurred as a result of the shooting, as well as Glen’s role in causing her injuries. However, it was not improper to take those factors into account in the context of the divorce action. As we explained in Walther, 709 P.2d at 388 (citing Anderson v. Anderson, 104 Utah 104, 109, 138 P.2d 252, 254 (1943)), “[I]njuries and attendant medical expenses [caused by a spouse’s tort] may be considered” in deciding the level of need of the other spouse in a divorce proceeding. And because Elaine’s counterclaim for divorce was based on Glen’s cruelty to her, it was proper for Judge Tibbs to consider the issue of Glen’s fault in causing those injuries. Merely because Judge Tibbs considered facts relevant to the divorce action that were also relevant to the tort action does not mean that he impermissibly adjudicated the tort claims in the divorce action.
 To do otherwise may raise significant concerns if a fact question with respect to which a party has requested a jury and is entitled to a jury verdict is first decided by a judge in an equitable proceeding. Cf. Beacon Theaters, Inc. v. Westover, 359 U.S. 500, 510–11, 79 S.Ct. 948, 956–957, 3 L.Ed.2d 988 (1959) (the federal constitutional right to a jury trial of legal issues may be lost through prior judicial determination of equitable claims only in the most imperative circumstances). However, those concerns have not been raised by the parties to this case.
In the case of Walther v. Walther (709 P.2d 387 (1985)), the Utah Supreme Court opined:
In Lord v. Shaw, Utah, 665 P.2d 1288 (1983), we observed:
[T]he trial court held that the plaintiff was barred by res judicata from suing her ex-husband for torts which occurred during the marriage, because his liability for any tort should have been litigated in the divorce action. We do not comment on this ruling other than to observe that actionable torts between married persons should not be litigated in a divorce proceeding. We believe that divorce actions will become unduly complicated in their trial and disposition if torts can be or must be litigated in the same action. A divorce action is highly equitable in nature, whereas the trial of a tort claim is at law and may well involve, as in this case, a request for trial by jury. The administration of justice will be better served by keeping the two proceedings separate.
Id., 665 P.2d at 1291 (citation omitted). The trial court should not have tried the wife’s tort claim as part of this divorce action. That part of the award is therefore vacated.
Utah Family Law, LC | divorceutah.com | 801-466-9277
(11) Eric Johnson’s answer to Can you file for defamation while a custody battle is happening? – Quora