working ha

DISTRIBUTED_SETUP.md

@@ -0,0 +1,236 @@
+# Distributed Phoenix WebSocket Game
+
+This project demonstrates a distributed Phoenix application with automatic failover using native Erlang clustering.
+
+## Architecture
+
+- **3 Phoenix Nodes**: Running in Docker containers, forming a distributed Erlang cluster
+- **Global Process Registry**: Uses `:global` to ensure only one `GameState` GenServer runs across the cluster
+- **Automatic Failover**: If a node goes down, another node automatically takes over the GameState
+- **Nginx Load Balancer**: Routes WebSocket connections to healthy nodes
+- **Client Failover**: Frontend automatically switches to another server if connection is lost
+
+## How It Works
+
+### Distributed Erlang Clustering
+
+- Each Phoenix container starts with a unique node name (e.g., `backend@phoenix1`)
+- All nodes share the same Erlang cookie for authentication
+- `Backend.Cluster` module automatically connects nodes on startup
+- Nodes use EPMD (Erlang Port Mapper Daemon) for discovery
+
+### Singleton Game State
+
+- `Backend.GameState` is registered globally using `{:global, __MODULE__}`
+- Only ONE instance runs across all nodes at any time
+- If the node running GameState crashes, Erlang automatically starts it on another node
+- All nodes can communicate with the GameState regardless of where it's running
+
+### Client Failover
+
+- Frontend maintains a list of all backend servers
+- Automatically reconnects to the next server if connection fails
+- Uses exponential backoff and retry logic
+- Displays current connection status
+
+## Setup
+
+### Prerequisites
+
+- Docker and Docker Compose
+- Or: Elixir 1.15+, Erlang 26+, Node.js 18+
+
+### Running with Docker
+
+1. Build and start all services:
+
+```bash
+docker-compose up --build
+```
+
+This starts:
+- `phoenix1` on port 4001
+- `phoenix2` on port 4002
+- `phoenix3` on port 4003
+- `nginx` load balancer on port 4000
+
+2. Open the client (in a separate terminal):
+
+```bash
+cd client
+pnpm install
+pnpm dev
+```
+
+3. Open http://localhost:5173 in your browser
+
+### Running Locally (Development)
+
+Terminal 1 - Backend Node 1:
+```bash
+cd backend
+mix deps.get
+export RELEASE_NODE=backend@127.0.0.1
+export RELEASE_COOKIE=mycookie
+export PORT=4001
+export CLUSTER_NODES="backend@127.0.0.1"
+iex --name backend@127.0.0.1 --cookie mycookie -S mix phx.server
+```
+
+Terminal 2 - Backend Node 2:
+```bash
+cd backend
+export RELEASE_NODE=backend@127.0.0.2
+export RELEASE_COOKIE=mycookie
+export PORT=4002
+export CLUSTER_NODES="backend@127.0.0.1,backend@127.0.0.2"
+iex --name backend@127.0.0.2 --cookie mycookie -S mix phx.server
+```
+
+Terminal 3 - Frontend:
+```bash
+cd client
+pnpm install
+pnpm dev
+```
+
+## Testing Failover
+
+### Test 1: Stop a node
+
+```bash
+# Stop one container
+docker-compose stop phoenix1
+
+# The game continues running on phoenix2 or phoenix3
+# Clients automatically reconnect to available nodes
+```
+
+### Test 2: Kill the node running GameState
+
+1. Find which node is running GameState:
+```bash
+docker-compose exec phoenix1 /app/bin/backend remote
+# In the IEx shell:
+:global.whereis_name(Backend.GameState)
+# This shows {pid, node_name}
+```
+
+2. Stop that specific node:
+```bash
+docker-compose stop phoenix2  # or whichever node is running it
+```
+
+3. The GameState automatically starts on another node
+4. All players remain in the game
+
+### Test 3: Network partition
+
+```bash
+# Disconnect a node from the network
+docker network disconnect websocket-testing_app_net phoenix3
+
+# Reconnect it
+docker network connect websocket-testing_app_net phoenix3
+```
+
+## Monitoring the Cluster
+
+### Check connected nodes
+
+```bash
+docker-compose exec phoenix1 /app/bin/backend remote
+```
+
+In the IEx shell:
+```elixir
+# List all connected nodes
+Node.list()
+
+# Check which node is running GameState
+:global.whereis_name(Backend.GameState)
+
+# Get current game state
+Backend.GameState.get_state()
+
+# Check registered global processes
+:global.registered_names()
+```
+
+### View logs
+
+```bash
+# All containers
+docker-compose logs -f
+
+# Specific container
+docker-compose logs -f phoenix1
+```
+
+## Configuration
+
+### Environment Variables
+
+- `RELEASE_NODE`: Node name (e.g., `backend@phoenix1`)
+- `RELEASE_COOKIE`: Erlang cookie for cluster authentication
+- `CLUSTER_NODES`: Comma-separated list of nodes to connect to
+- `PORT`: HTTP port for Phoenix endpoint
+- `SECRET_KEY_BASE`: Phoenix secret key
+
+### Scaling
+
+To add more nodes, edit `docker-compose.yml`:
+
+```yaml
+phoenix4:
+  # Same config as phoenix1-3, with unique:
+  # - container_name: phoenix4
+  # - hostname: phoenix4
+  # - RELEASE_NODE: backend@phoenix4
+  # - ports: "4004:4000"
+  # - ipv4_address: 172.25.0.14
+```
+
+Update `CLUSTER_NODES` in all services to include `backend@phoenix4`.
+
+## How to Play
+
+- Use **WASD** keys to move your player
+- Your player is shown in red, others in blue
+- The game state is shared across all nodes
+- Try killing nodes to see failover in action!
+
+## Troubleshooting
+
+### Nodes not connecting
+
+1. Check all nodes have the same `RELEASE_COOKIE`
+2. Verify EPMD is running: `docker-compose exec phoenix1 epmd -names`
+3. Check firewall allows ports 4369 (EPMD) and 9000-9100 (distributed Erlang)
+
+### GameState not starting
+
+1. Check logs: `docker-compose logs -f`
+2. Verify only one instance exists globally: `:global.registered_names()`
+3. Restart all nodes: `docker-compose restart`
+
+### Frontend not connecting
+
+1. Check nginx is running: `docker-compose ps nginx`
+2. Verify at least one Phoenix node is healthy
+3. Check browser console for connection errors
+4. Try connecting directly to a node: http://localhost:4001
+
+## Production Considerations
+
+- **Change the Erlang cookie**: Use a strong secret
+- **Use proper SSL/TLS**: Configure HTTPS for WebSocket connections
+- **Add health checks**: Monitor node health and GameState availability
+- **Persistent storage**: Add database for game state persistence
+- **Rate limiting**: Protect against abuse
+- **Monitoring**: Add Prometheus/Grafana for metrics
+- **Logging**: Centralize logs with ELK or similar
+
+## License
+
+MIT

client/.dockerignore

@@ -0,0 +1,27 @@
+# Dependencies
+node_modules
+
+# Build outputs
+dist
+dist-ssr
+*.local
+
+# Editor directories and files
+.vscode/*
+!.vscode/extensions.json
+.idea
+.DS_Store
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?
+
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+lerna-debug.log*

client/Dockerfile

@@ -0,0 +1,21 @@
+# Build stage
+FROM node:20-alpine AS build
+
+WORKDIR /app
+
+COPY package.json pnpm-lock.yaml ./
+
+RUN npm install -g pnpm && pnpm install --frozen-lockfile
+
+COPY . .
+
+RUN pnpm build
+
+FROM nginx:alpine
+
+COPY --from=build /app/dist /usr/share/nginx/html
+
+COPY nginx.conf /etc/nginx/conf.d/default.conf
+
+EXPOSE 80
+CMD ["nginx", "-g", "daemon off;"]

client/nginx.conf

@@ -0,0 +1,22 @@
+server {
+    listen 80;
+    server_name localhost;
+    root /usr/share/nginx/html;
+    index index.html;
+
+    # Enable gzip compression
+    gzip on;
+    gzip_vary on;
+    gzip_min_length 1024;
+    gzip_types text/plain text/css text/xml text/javascript application/javascript application/xml+rss application/json;
+
+    location / {
+        try_files $uri $uri/ /index.html;
+    }
+
+    # Cache static assets
+    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
+        expires 1y;
+        add_header Cache-Control "public, immutable";
+    }
+}

client/package.json

@@ -10,12 +10,14 @@
     "preview": "vite preview"
   },
   "dependencies": {
+    "phoenix": "^1.8.4",
     "react": "^19.2.0",
     "react-dom": "^19.2.0"
   },
   "devDependencies": {
     "@eslint/js": "^9.39.1",
     "@types/node": "^24.10.1",
+    "@types/phoenix": "^1.6.7",
     "@types/react": "^19.2.7",
     "@types/react-dom": "^19.2.3",
     "@vitejs/plugin-react": "^5.1.1",

client/pnpm-lock.yaml

@@ -8,6 +8,9 @@ importers:
 
   .:
     dependencies:
+      phoenix:
+        specifier: ^1.8.4
+        version: 1.8.4
       react:
         specifier: ^19.2.0
         version: 19.2.4
@@ -21,6 +24,9 @@ importers:
       '@types/node':
         specifier: ^24.10.1
         version: 24.10.13
+      '@types/phoenix':
+        specifier: ^1.6.7
+        version: 1.6.7
       '@types/react':
         specifier: ^19.2.7
         version: 19.2.14
@@ -515,6 +521,9 @@ packages:
   '@types/node@24.10.13':
     resolution: {integrity: sha512-oH72nZRfDv9lADUBSo104Aq7gPHpQZc4BTx38r9xf9pg5LfP6EzSyH2n7qFmmxRQXh7YlUXODcYsg6PuTDSxGg==}
 
+  '@types/phoenix@1.6.7':
+    resolution: {integrity: sha512-oN9ive//QSBkf19rfDv45M7eZPi0eEXylht2OLEXicu5b4KoQ1OzXIw+xDSGWxSxe1JmepRR/ZH283vsu518/Q==}
+
   '@types/react-dom@19.2.3':
     resolution: {integrity: sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==}
     peerDependencies:
@@ -930,6 +939,9 @@ packages:
     resolution: {integrity: sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==}
     engines: {node: '>=8'}
 
+  phoenix@1.8.4:
+    resolution: {integrity: sha512-5wdO9mqU4l0AmcexN8mA0m1BsC4ROv2l55MN31gbxmJPzghc4SxVxnNTh9qaummYa2pbwkoBvG8FezO7cjdr8A==}
+
   picocolors@1.1.1:
     resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
 
@@ -1480,6 +1492,8 @@ snapshots:
     dependencies:
       undici-types: 7.16.0
 
+  '@types/phoenix@1.6.7': {}
+
   '@types/react-dom@19.2.3(@types/react@19.2.14)':
     dependencies:
       '@types/react': 19.2.14
@@ -1930,6 +1944,8 @@ snapshots:
 
   path-key@3.1.1: {}
 
+  phoenix@1.8.4: {}
+
   picocolors@1.1.1: {}
 
   picomatch@4.0.3: {}

client/src/App.css

@@ -1,8 +1,6 @@
 #root {
-  max-width: 1280px;
+  margin: 0;
-  margin: 0 auto;
+  padding: 0;
-  padding: 2rem;
-  text-align: center;
 }
 
 .logo {

client/src/App.tsx

@@ -1,35 +1,232 @@
-import { useState } from 'react'
+import { useEffect, useRef, useState } from "react";
-import reactLogo from './assets/react.svg'
+import { Socket, Channel } from "phoenix";
-import viteLogo from '/vite.svg'
+import "./App.css";
-import './App.css'
 
-function App() {
+interface Player {
-  const [count, setCount] = useState(0)
+  x: number;
-
+  y: number;
-  return (
-    <>
-      <div>
-        <a href="https://vite.dev" target="_blank">
-          <img src={viteLogo} className="logo" alt="Vite logo" />
-        </a>
-        <a href="https://react.dev" target="_blank">
-          <img src={reactLogo} className="logo react" alt="React logo" />
-        </a>
-      </div>
-      <h1>Vite + React</h1>
-      <div className="card">
-        <button onClick={() => setCount((count) => count + 1)}>
-          count is {count}
-        </button>
-        <p>
-          Edit <code>src/App.tsx</code> and save to test HMR
-        </p>
-      </div>
-      <p className="read-the-docs">
-        Click on the Vite and React logos to learn more
-      </p>
-    </>
-  )
 }
 
-export default App
+interface GameState {
+  [playerId: string]: Player;
+}
+
+// List of WebSocket servers - we'll connect to all of them
+const WS_SERVERS = [
+  "ws://localhost:4001/socket",
+  "ws://localhost:4002/socket",
+  "ws://localhost:4003/socket",
+];
+
+function App() {
+  const [players, setPlayers] = useState<GameState>({});
+  const [myPlayerId, setMyPlayerId] = useState<string | null>(null);
+  const [connectionStatus, setConnectionStatus] =
+    useState<string>("connecting");
+  const socketsRef = useRef<Socket[]>([]);
+  const channelsRef = useRef<Channel[]>([]);
+  const keysPressed = useRef<Set<string>>(new Set());
+
+  useEffect(() => {
+    // Connect to all servers concurrently
+    const sockets = WS_SERVERS.map((serverUrl) => {
+      console.log(`Connecting to ${serverUrl}`);
+
+      const socket = new Socket(serverUrl, {
+        timeout: 3000,
+        reconnectAfterMs: () => 2000, // Keep trying to reconnect
+      });
+
+      // Handle connection events
+      socket.onOpen(() => {
+        console.log(`✓ Connected to ${serverUrl}`);
+        updateConnectionStatus();
+      });
+
+      socket.onError((error) => {
+        console.error(`✗ Error on ${serverUrl}:`, error);
+        updateConnectionStatus();
+      });
+
+      socket.onClose(() => {
+        console.log(`✗ Disconnected from ${serverUrl}`);
+        updateConnectionStatus();
+      });
+
+      socket.connect();
+      return socket;
+    });
+
+    socketsRef.current = sockets;
+
+    // Join game channel on all connected sockets
+    const channels = sockets.map((socket, index) => {
+      const channel = socket.channel("game:lobby", {});
+
+      channel
+        .join()
+        .receive("ok", () => {
+          console.log(`✓ Joined channel on ${WS_SERVERS[index]}`);
+          updateConnectionStatus();
+        })
+        .receive("error", (resp) => {
+          console.log(`✗ Failed to join on ${WS_SERVERS[index]}:`, resp);
+        })
+        .receive("timeout", () => {
+          console.log(`✗ Timeout joining on ${WS_SERVERS[index]}`);
+        });
+
+      // Listen for game state updates from any server
+      channel.on("game_state", (payload: { players: GameState }) => {
+        setPlayers(payload.players);
+
+        // Set our player ID from the first state update if not set
+        if (!myPlayerId && Object.keys(payload.players).length > 0) {
+          const playerIds = Object.keys(payload.players);
+          if (playerIds.length > 0) {
+            setMyPlayerId(playerIds[playerIds.length - 1]);
+          }
+        }
+      });
+
+      return channel;
+    });
+
+    channelsRef.current = channels;
+
+    const updateConnectionStatus = () => {
+      const joined = channels.filter((c) => c.state === "joined").length;
+      setConnectionStatus(`${joined}/${WS_SERVERS.length} servers active`);
+    };
+
+    // Periodic status update
+    const statusInterval = setInterval(updateConnectionStatus, 1000);
+
+    // Cleanup on unmount
+    return () => {
+      clearInterval(statusInterval);
+      channels.forEach((channel) => channel.leave());
+      sockets.forEach((socket) => socket.disconnect());
+    };
+  }, [myPlayerId]);
+
+  // Handle keyboard input - send to first available channel
+  useEffect(() => {
+    const handleKeyDown = (e: KeyboardEvent) => {
+      const key = e.key.toLowerCase();
+      if (["w", "a", "s", "d"].includes(key)) {
+        e.preventDefault();
+
+        // Only send if not already pressed (prevent repeat)
+        if (!keysPressed.current.has(key)) {
+          keysPressed.current.add(key);
+
+          // Send to first joined channel (they all share same game state)
+          const activeChannel = channelsRef.current.find(
+            (c) => c.state === "joined",
+          );
+          if (activeChannel) {
+            activeChannel.push("move", { direction: key });
+          }
+        }
+      }
+    };
+
+    const handleKeyUp = (e: KeyboardEvent) => {
+      const key = e.key.toLowerCase();
+      if (["w", "a", "s", "d"].includes(key)) {
+        keysPressed.current.delete(key);
+      }
+    };
+
+    window.addEventListener("keydown", handleKeyDown);
+    window.addEventListener("keyup", handleKeyUp);
+
+    return () => {
+      window.removeEventListener("keydown", handleKeyDown);
+      window.removeEventListener("keyup", handleKeyUp);
+    };
+  }, []);
+
+  return (
+    <div
+      style={{
+        width: "100vw",
+        height: "100vh",
+        background: "#1a1a2e",
+        overflow: "hidden",
+        position: "relative",
+      }}
+    >
+      {/* Connection status */}
+      <div
+        style={{
+          position: "absolute",
+          top: 10,
+          right: 10,
+          color: "white",
+          fontFamily: "monospace",
+          background: "rgba(0,0,0,0.5)",
+          padding: "10px",
+          borderRadius: "5px",
+          fontSize: "12px",
+        }}
+      >
+        <div>Status: {connectionStatus}</div>
+        <div>Players: {Object.keys(players).length}</div>
+      </div>
+
+      {/* Game canvas */}
+      <div
+        style={{
+          position: "relative",
+          width: "800px",
+          height: "600px",
+          background: "#16213e",
+          margin: "50px auto",
+          border: "2px solid #0f3460",
+          overflow: "hidden",
+        }}
+      >
+        {Object.entries(players).map(([id, player]) => (
+          <div
+            key={id}
+            style={{
+              position: "absolute",
+              left: player.x,
+              top: player.y,
+              width: "20px",
+              height: "20px",
+              borderRadius: "50%",
+              background: id === myPlayerId ? "#e94560" : "#53a8b6",
+              border:
+                id === myPlayerId ? "3px solid #ff6b6b" : "2px solid #48d6e0",
+              transition: "all 0.1s linear",
+              transform: "translate(-50%, -50%)",
+              boxShadow:
+                id === myPlayerId ? "0 0 10px #e94560" : "0 0 5px #53a8b6",
+            }}
+          >
+            {id === myPlayerId && (
+              <div
+                style={{
+                  position: "absolute",
+                  top: "-25px",
+                  left: "50%",
+                  transform: "translateX(-50%)",
+                  color: "#fff",
+                  fontSize: "10px",
+                  whiteSpace: "nowrap",
+                }}
+              >
+                You
+              </div>
+            )}
+          </div>
+        ))}
+      </div>
+    </div>
+  );
+}
+
+export default App;

client/src/index.css

@@ -24,8 +24,6 @@ a:hover {
 
 body {
   margin: 0;
-  display: flex;
-  place-items: center;
   min-width: 320px;
   min-height: 100vh;
 }

docker-compose.yml

@@ -0,0 +1,82 @@
+version: "3.8"
+
+services:
+  phoenix1:
+    build:
+      context: ./backend
+      dockerfile: Dockerfile
+    container_name: phoenix1
+    hostname: phoenix1
+    environment:
+      - RELEASE_NODE=backend@phoenix1
+      - RELEASE_DISTRIBUTION=sname
+      - RELEASE_COOKIE=super_secret_cookie_change_in_production
+      - PHX_HOST=localhost
+      - PHX_SERVER=true
+      - PORT=4000
+      - DATABASE_URL=ecto://postgres:postgres@db/backend_dev
+      - SECRET_KEY_BASE=W8nGKNhNR8vKj6A4VnwN5h5h7RZvkKmZPqxqzLzYxXGQqC6HnKp2Wm8MNqKpQxZv
+      - CLUSTER_NODES=backend@phoenix1,backend@phoenix2,backend@phoenix3
+    ports:
+      - "4001:4000"
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://localhost:4000/api/health"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  phoenix2:
+    build:
+      context: ./backend
+      dockerfile: Dockerfile
+    container_name: phoenix2
+    hostname: phoenix2
+    environment:
+      - RELEASE_NODE=backend@phoenix2
+      - RELEASE_DISTRIBUTION=sname
+      - RELEASE_COOKIE=super_secret_cookie_change_in_production
+      - PHX_HOST=localhost
+      - PHX_SERVER=true
+      - PORT=4000
+      - DATABASE_URL=ecto://postgres:postgres@db/backend_dev
+      - SECRET_KEY_BASE=W8nGKNhNR8vKj6A4VnwN5h5h7RZvkKmZPqxqzLzYxXGQqC6HnKp2Wm8MNqKpQxZv
+      - CLUSTER_NODES=backend@phoenix1,backend@phoenix2,backend@phoenix3
+    ports:
+      - "4002:4000"
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://localhost:4000/api/health"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  phoenix3:
+    build:
+      context: ./backend
+      dockerfile: Dockerfile
+    container_name: phoenix3
+    hostname: phoenix3
+    environment:
+      - RELEASE_NODE=backend@phoenix3
+      - RELEASE_DISTRIBUTION=sname
+      - RELEASE_COOKIE=super_secret_cookie_change_in_production
+      - PHX_HOST=localhost
+      - PHX_SERVER=true
+      - PORT=4000
+      - DATABASE_URL=ecto://postgres:postgres@db/backend_dev
+      - SECRET_KEY_BASE=W8nGKNhNR8vKj6A4VnwN5h5h7RZvkKmZPqxqzLzYxXGQqC6HnKp2Wm8MNqKpQxZv
+      - CLUSTER_NODES=backend@phoenix1,backend@phoenix2,backend@phoenix3
+    ports:
+      - "4003:4000"
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://localhost:4000/api/health"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  client:
+    build:
+      context: ./client
+      dockerfile: Dockerfile
+    container_name: client
+    ports:
+      - "5173:80"

nginx.conf

@@ -0,0 +1,48 @@
+events {
+    worker_connections 1024;
+}
+
+http {
+    upstream phoenix_backend {
+        least_conn;
+        server phoenix1:4000 max_fails=3 fail_timeout=30s;
+        server phoenix2:4000 max_fails=3 fail_timeout=30s;
+        server phoenix3:4000 max_fails=3 fail_timeout=30s;
+    }
+
+    # Map to track which backend server a WebSocket connection should use
+    map $http_upgrade $connection_upgrade {
+        default upgrade;
+        '' close;
+    }
+
+    server {
+        listen 4000;
+        server_name localhost;
+
+        # WebSocket configuration
+        location /socket {
+            proxy_pass http://phoenix_backend;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection $connection_upgrade;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            
+            # WebSocket timeout settings
+            proxy_read_timeout 86400;
+            proxy_send_timeout 86400;
+        }
+
+        # HTTP fallback for health checks
+        location / {
+            proxy_pass http://phoenix_backend;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+        }
+    }
+}