Multi-Agent Systems: Hierarchies, Collaboration, and Swarms

As business workflows grow complex, forcing a single monolithic agent to handle everything fails:

• Its system prompt becomes 2,000 words long with dozens of conflicting instructions.
• It must hold 30 different tool definitions in its context window.
• If it makes an error in step 2, it propagates confusion through the rest of the workflow.

The modern architectural solution is Multi-Agent Systems: decomposing complex workflows into a team of specialized, cooperating agents.

1. The 3 Multi-Agent Collaboration Patterns

flowchart TD
    subgraph Pattern1["1. Hierarchical (Supervisor / Orchestrator)"]
        Sup["Supervisor Agent<br/>(Routes tasks, evaluates results)"]
        W1["SQL Specialist Agent<br/>(Only queries DB)"]
        W2["PySpark Specialist Agent<br/>(Only writes transformations)"]
        W3["Data Governance Agent<br/>(Only checks PII & compliance)"]

        Sup --> W1
        Sup --> W2
        Sup --> W3
    end

    subgraph Pattern2["2. Sequential Assembly Line (Pipeline)"]
        A1["Analyst Agent<br/>Decomposes business query"] --> A2["Developer Agent<br/>Writes PySpark ETL"]
        A2 --> A3["QA / Reviewer Agent<br/>Audits for data skew & memory leaks"]
    end

    subgraph Pattern3["3. Collaborative Debate / Peer Review"]
        Gen["SQL Generator Agent<br/>Proposes query"] <--> Aud["SQL Auditor Agent<br/>Validates index usage & partition pruning"]
    end

    style Sup fill:#7c3aed,color:#fff
    style W1 fill:#2563eb,color:#fff
    style W2 fill:#059669,color:#fff
    style W3 fill:#d97706,color:#fff
    style A1 fill:#1e293b,color:#fff
    style A2 fill:#1e293b,color:#fff
    style A3 fill:#1e293b,color:#fff

2. Building a Multi-Agent Supervisor Pattern

In a Hierarchical Supervisor system:

1. The Supervisor receives the high-level user request.
2. The Supervisor chooses which specialist to invoke next (or decides the task is FINISHED).
3. Each specialist agent runs in isolation with only the tools it needs.
4. The output is routed back to the Supervisor to determine the next step.

from typing import Dict, Any

class SQLSpecialist:
    def run(self, task: str) -> str:
        return "SQL Specialist Output: Generated query 'SELECT date, sum(revenue) FROM fct_sales GROUP BY 1;'"

class SparkSpecialist:
    def run(self, task: str) -> str:
        return "Spark Specialist Output: Generated PySpark code 'df.groupBy("date").agg(sum("revenue"))'"

class MultiAgentSupervisor:
    def __init__(self):
        self.sql_agent = SQLSpecialist()
        self.spark_agent = SparkSpecialist()

    def orchestrate(self, user_goal: str) -> str:
        print(f"[Supervisor] Received goal: '{user_goal}'")

        # Simulated routing decision
        if "sql" in user_goal.lower() or "warehouse" in user_goal.lower():
            print("[Supervisor] Routing task to SQL Specialist...")
            return self.sql_agent.run(user_goal)
        elif "spark" in user_goal.lower() or "dataframe" in user_goal.lower():
            print("[Supervisor] Routing task to PySpark Specialist...")
            return self.spark_agent.run(user_goal)
        else:
            return "Task completed."

if __name__ == "__main__":
    supervisor = MultiAgentSupervisor()
    result = supervisor.orchestrate("Write a Spark DataFrame aggregation for daily revenue")
    print("
Result:", result)

Key Takeaways

• Specialized agents outperform generalist agents by focusing on narrow prompts and small toolsets.
• Hierarchical Supervisors dynamically delegate work and synthesize sub-agent outputs.
• Peer review / debate patterns prevent bad code or unoptimized SQL from reaching production.