Robots Txt

100 pts. Web, basic. CWA 2026, challenge 03. Target: https://ch-003.ctf.africa.

A corporate portal attempts to hide restricted administrative paths by defining crawler exclusion standards without enforcing endpoint authentication. Inspect the spider directives to discover the unlinked directory.

First look

Landing page is a bare SimpleHTTP stub, no links, no forms:

class="hljs-keyword"><html>class="hljs-keyword"><head>class="hljs-keyword"><title>003-web-03-robots-txtclass="hljs-keyword"></title>class="hljs-keyword"></head>class="hljs-keyword"><body>
class="hljs-keyword"><h2>Cyberweek Africa CTF Target: 003-web-03-robots-txtclass="hljs-keyword"></h2>class="hljs-keyword"><p>Endpoint operational.class="hljs-keyword"></p>class="hljs-keyword"></body>class="hljs-keyword"></html>

Server is SimpleHTTP/0.6 Python/3.11.16 over HTTP/2. Port 80 is filtered, only 443 answers, so HTTPS is the only ingress. There is nothing to click, which is the substance of the challenge: the interesting content is not linked from anywhere.

The robots.txt step

The intended read is /robots.txt, which on a well-behaved portal lists the paths the site tries to keep out of search engines. Here it is a genuine 404:

ffcurl -i https://ch-003.ctf.africa/robots.txt
HTTP/2 404
content-type: text/html;charset=utf-8
server: SimpleHTTP/0.6 Python/3.11.16

That is the exposure the description names outright: the site declares exclusion standards but never enforces authentication on the endpoints behind them. The directives are advisory, so the protected path is still reachable on a guess.

The surface was confirmed minimal before guessing. A wordlist sweep of the root returned exactly one live file:

ffuf -u "https://ch-003.ctf.africa/FUZZ" -w /usr/share/seclists/Discovery/Web-Content/common.txt \
     -mc 200,204,301,302,307,401,403,500 -t 40 -timeout 8 -s

index.html

No directories returned 403 either, so there was no listing to walk. The administrative path is unlinked and unenforced, which leaves a short list of conventional guesses.

The unlinked path

/flag.txt is the file behind the phantom exclusion rule, and it answers without any authentication:

ffcurl -i https://ch-003.ctf.africa/flag.txt
HTTP/2 200
content-type: text/plain
content-length: 31

FLAG{cwa2026_0330fc52528e15d7}

Nothing in the response indicates the request was privileged. Same headers, same origin, no cookie, no token, which is the whole lesson of the challenge. Exclusion directives are a request to crawlers, not a control. Anything that gates an administrative path needs authentication in front of it.

Takeaway

A robots.txt is a courtesy to well-behaved crawlers and nothing more. Reading one indicates where a site keeps its interesting paths, and a real control has to reject unauthorised requests on its own. When a portal hides a directory only in its spider directives, the conventional administrative filenames are worth trying directly.

Flag:

FLAG{cwa2026_0330fc52528e15d7}