AI Agents with Open-Source LLMs: Practical Integration with the Model Context Protocol (MCP)

Open-source LLMs remain essential for several reasons, technical, strategic, ethical, and economic. They enable fine-tuning for specific purposes and domains, and also allow execution in isolated environments, which makes it possible to securely handle sensitive or confidential data. Because they are more accessible, they also foster innovation by making adoption easier for smaller companies and organizations.

However, open-source LLMs have important limitations: they are not updated with recent events, they do not have access to private data, and they generally lack native integration with real-world systems and software. Until recently, integrating tools with an open-source LLM, such as LLaMA (Meta) or Mistral, required developing a custom integration layer for each model, with low potential for reuse.

LLM access to private data, whether through a database or a file system, can help solve everyday problems within a company, such as the following:

• HR and People Management: Automatically query employee information (e.g., overdue medical exams, accrued vacation days) in a Postgres database and generate reports for the Human Resources department.
• Finance: Create dashboards that fetch real-time cash flow data from a MongoDB database, allowing managers to view balance projections based on already recorded inflows and outflows, provided those projections are already stored in the database.
• Customer Support: Query ticket history data in Postgres so that an AI agent can suggest quick responses to the support team.
• Compliance and Auditing: An AI agent can scan internal directories, locate contract files, extract critical clauses (e.g., deadlines or penalties), and consolidate everything into an automated report.
• Engineering: Search for technical documents in PDF or Word format in shared folders to speed up the preparation of proposals or project reports.
• Legal: Locate filings and attachments within the file system to prepare procedural responses.

The Model Context Protocol (MCP), an open protocol introduced by Anthropic in 2024, aims to standardize how LLMs interact with external data and tools, making it easier to develop more efficient and interoperable AI agents. An AI agent is essentially an autonomous software system that executes task workflows to achieve a goal on behalf of the user.

As the ecosystem of MCP-based tools matures, it is already possible to integrate:

• Access to public information from the internet, via APIs (e.g., Perplexity.ai MCP, Pupeteer MCP).
• Access to private data, with integration to databases (e.g., MongoDB MCP, Postgres MCP), file systems, cloud storage services, and email servers.
• Location services: Integration with mapping services (e.g., Google Maps MCP) and weather forecasting (e.g., weather-mcp)
• Integration with software development and monitoring tools, such as version control (e.g., Github MCP, GitLab MCP) and containerization (e.g., Docker MCP), API development (e.g., Postman MCP), and monitoring (e.g., Grafana MCP).
• Integration with cloud services, such as (GoogleCloud Run MCP, AWS MCP, Azure MCP).
• Integration with industrial systems: such as OPCUA-MCP

We created a demo project, available here, based on the Fast MCP Python library. This library allows users to quickly create their own MCP server. The project is configured to provide the LLM with a set of tools, including:

• Multiple tools that query data on scientific publications through the public API of OpenAlex.org, and
• A tool to download a container image from Docker Hub, using credentials provided by the user in the .env and .gemini/settings.json files.

The server can be accessed by any LLM client that supports MCPs. However, it has been tested and validated with Gemini CLI, whose configuration is included in the .gemini folder of the demo project. Below are two example prompts that use the integrated tools: one to access external APIs and another to download images from the Docker repository. Installation and configuration procedures are detailed in the README.md file of the Git repository.

Prompt 1 – Number of publications from Japan:

What is the number of publication sources in Japan?

Prompt 2 – Distribution of publication source types in Brazil:

Show me the distribution of the types of publication sources in Brazil.

Prompt 3 – Access DockerHub and download the Python image:

Use the mainAPIServer MCP to pull the python:3.9-slim image from Docker Hub and run it to display the Python version. Ensure Docker Hub authentication uses the credentials provided in .gemini/settings.json and .env.