Logs

This section provides a detailed technical overview of logging in the Aindo Synthetic Data Platform.

The Aindo application is deployed on Kubernetes, and all resources write log information to standard output. When the application is deployed on cloud providers using a managed Kubernetes service, logs are typically redirected automatically to the provider’s logging console (for example, Cloud Logging in Google Cloud Platform or CloudWatch Logs in AWS). For self-managed or self-hosted clusters, logs can be collected using standard log collection tools.

Log Format

Logs of the API deployment are structured as follows:

<timestamp> [<request_id>][<correlation_id>] <level> --- <source> --- <message>

The following is a detailed description of each field included in the above log format:

<timestamp>: The date and time when the log entry was created, in ISO 8601 format with milliseconds (YYYY-MM-DDTHH:MM:SS.mmm).
<request_id>: A unique identifier assigned to a single incoming request. Used to trace all log entries generated while that request is being processed.
<correlation_id>: An identifier that groups multiple related requests or operations together (e.g. in a distributed system or multi-service workflow). Useful for tracing an end-to-end flow across services.
<level>: The severity of the log entry.
<source>: The name of the component, class or module that generated the log entry.
<message>: The actual log message or content describing what happened. May include structured information or contextual details.

Possible values for <level> and their meanings are:

DEBUG: Detailed information mainly useful for diagnosing problems during development or troubleshooting.
INFO: General operational messages indicating that things are working as expected.
WARNING: An indication of a potential problem or important situation that does not stop the program from working.
ERROR: An issue that has occurred, preventing some functionality from completing successfully.
CRITICAL: A severe problem indicating that the system or a major component may be unable to continue running.

The frontend deployment uses the standard Nginx “combined” log format:

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"

Below is a breakdown of what each field represents:

$remote_addr: client address
$remote_user: username supplied with the Basic authentication
$time_local: local time in the Common Log Format
$request: full original request line
$status: response status
$body_bytes_sent: number of bytes sent to a client, not counting the response header
$http_referer: URL of the webpage that linked to the requested resource, sent in the HTTP Referer header
$http_user_agent: User-Agent string sent in the HTTP request header

Exception Logging

When an exception occurs in the API or worker deployments, the main log message follows the defined format, including timestamp, level, request/correlation IDs, and source. The full traceback or error details are appended immediately after as multi-line output and are not formatted according to the log format. This is an example:

2025-08-20T16:25:00.722 [b33f803f65439300703b04dc29634d12][-] INFO --- app --- Started POST /api/v1/catalogs HTTP/1.1 from 127.0.0.1
2025-08-20T16:25:00.758 [b33f803f65439300703b04dc29634d12][-] ERROR --- app --- Generic exception
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 111, in receive
    return self.receive_nowait()
  File "/opt/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 106, in receive_nowait
    raise WouldBlock
anyio.WouldBlock
[...]
  File "/app/api/api/features.py", line 18, in _enabled
    feature = getattr(config.features, key)
  File "/opt/venv/lib/python3.10/site-packages/pydantic/main.py", line 891, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'FeaturesConfig' object has no attribute 'sources'
2025-08-20T16:25:00.761 [b33f803f65439300703b04dc29634d12][-] INFO --- app --- Completed 500 Internal Server Error in 38ms

Examples

HTTP request

When the HTTP API service receives a request, it generates at least the following log lines:

2025-03-20T13:24:26.898 [ca0c88f4f627937033181aeae24a5282][-] INFO --- app --- Started GET /api/v1/catalogs HTTP/1.1 from 172.100.0.19
2025-03-20T13:24:27.080 [ca0c88f4f627937033181aeae24a5282][-] INFO --- app --- Completed 200 OK in 182ms

The first line marks the beginning of the request, and the second line marks its completion. Additional logs may appear in between depending on processing.

Rate Limit Violation

If the API is called too frequently, or a specific action is attempted too many times, an HTTP error code 429 is returned. A log is generated at the WARNING level to indicate the rate limit violation. This is an example of a user who is repeatedly attempting to log in with the wrong password:

2025-03-20T13:56:01.455 [d7b75ec60d2f1a62ef11741a56fc481e][-] INFO --- app --- Started POST /api/v1/auth/flow/password/do HTTP/1.1 from 172.100.0.19
2025-03-20T13:56:01.464 [d7b75ec60d2f1a62ef11741a56fc481e][-] WARNING --- app --- Rate limit violation. Type: pwd, identity: 95hRfGm88p4x, exception: [H_429] Attempts are limited (Requests are being limited, wait a short period)
2025-03-20T13:56:01.465 [d7b75ec60d2f1a62ef11741a56fc481e][-] INFO --- app --- Completed 429 Too Many Requests in 10ms

Authentication failed

When a user attempts to log in with an incorrect password, the backend returns an HTTP 401 error, and a log is generated at the WARNING level to indicate the failed authentication attempt. Example:

2025-03-20T13:59:46.751 [b5bbf323be617f14984b94f57489dde1][-] INFO --- app --- Started POST /api/v1/auth/flow/password/do HTTP/1.1 from 172.100.0.19
2025-03-20T13:59:47.586 [b5bbf323be617f14984b94f57489dde1][-] WARNING --- app --- Authentication failed: wrong password. Identity: 95hRfGm88p4x
2025-03-20T13:59:47.586 [b5bbf323be617f14984b94f57489dde1][-] INFO --- app --- Completed 401 Unauthorized in 835ms