Marcos Oliveira Get in touch
Director / VP AI/ML Engineering · MBA · 20 years

I help teams move from AI ambition to production.

I give CTOs and founders the AI leadership of a hands-on VP — the strategy, the architecture, and the data to back it — turning agentic AI and MLOps into systems that ship and outcomes you can measure.

Agentic AI  ·  LLMOps  ·  MLOps  ·  Multi-Cloud (AWS + Azure)  ·  FinTech  ·  Team Leadership

Talk to me about your AI roadmap → See how I work →

Led AI/ML engineering and global delivery for Canada's largest financial institutions — driving −65% manual processing time with production AI.

VP AI/ML Engineering  ·  Block 64

A Canadian IT asset-intelligence platform — visibility across on-prem, cloud, SaaS & licensing.

Marcos Oliveira, Director / VP AI/ML Engineering
Marcos OliveiraDirector / VP AI/ML Engineering
About

For more than twenty years I've worked where engineering meets the decision — leading software and AI/ML organizations across SaaS and enterprise. Today I lead a cross-functional AI/ML team at Block 64, building across security, modern work and infrastructure.

Before that I spent over a decade at Lexmark Canada and Hewlett-Packard Canada, directing global managed-services delivery and full-lifecycle software rollouts for Canada's largest financial institutions — work that ran under the governance and internal-control standards those environments demand. I hold a double-degree MBA from the Rotman School of Management & Business School São Paulo (BSP) and a B.Sc. in Computer Science from Universidade Ibirapuera in Brazil. I hire and grow the best talent, build agile teams organically, and stay close to the detail — deep-diving with teams to raise standards and ship. I'm driven by collective success, I champion innovation and simpler processes, and I believe in continuous learning. A believer in social equity, I support United Way in the Greater Toronto Area.

Marcos Oliveira · Director / VP AI/ML Engineering · MBA, Rotman
What I stand by

Five values that drive how I lead and decide.

Leadership done right isn't about control — it's about clarity, connection and consistency. These are the principles I live by, and they show up in every decision, conversation and engagement.

I

Trust

A single source of truth — no filters, no spin. In an appraisal, a board meeting or a transaction, I give direct, honest feedback. Trust is built through consistency, not words.

II

Speed

If you don't act, your competitor will. Not rushing — moving with context, confidence and focus. Standing still is falling behind.

III

Growth

Education × innovation = growth — for a business and for a person. Learn by doing, stay curious, keep applying. The leaders who grow are the ones who never stop learning.

IV

Heart

Empathy and emotional intelligence aren't soft skills; they're leadership skills. Care about people, their development and their wellbeing — and they give you their best because they want to.

V

Unity

No goal is achieved alone. Whether building a team or deploying AI across a business, bring people with you, give them ownership, and make them part of the story.

Approach

Start with why. Prove it with data.

My leadership style follows Simon Sinek's "Start With Why" — leading with purpose, then making the impact with data, reporting and the evidence that influences the decision. Increasingly, that means leading the shift from the traditional SDLC to an Agentic Development Lifecycle (ADLC) — an AI-driven delivery model where agentic workflows speed execution without loosening governance.

The belief

A team should never have to guess. Good AI leadership replaces hype with evidence, and gives leaders the confidence to act on it.

The method

Start with the outcome that matters. Then bring the architecture, the data and the reporting that drive the decision — made clear enough for non-technical leaders to own.

The work

Director and VP-level AI leadership for CTOs and founders of mid-sized to large companies — across agentic AI, RAG, MLOps and responsible AI governance.

What I do

Six places I move the needle.

Engagements are scoped to the problem in front of you — an AI strategy, a production platform, an agentic system, or ongoing leadership. Each is something I've owned end to end.

I

AI-First strategy & roadmaps

Translating complex AI ambition into a technical roadmap aligned to business outcomes — revenue, retention and operational efficiency — with P&L ownership.

II

Agentic & generative AI

LLM integration, RAG pipelines, multi-agent systems and LLMOps with LangChain / LangGraph — built with runtime guardrails, not demos.

III

MLOps & AI/ML platforms

Feature stores, model serving and full CI/CD/CT pipelines with MLflow — cutting deployment cycles from weeks to days.

IV

Dual-cloud architecture

Hands-on AI/ML platforms across AWS (SageMaker, Bedrock) and Azure (ML Studio, Microsoft Foundry, AI Search), built for production scale.

V

Responsible AI & governance

Model observability, drift detection and SOC 2 Type II compliance across production AI — governance you can stand behind.

VI

Engineering leadership

Building and scaling high-performing AI/ML organizations (20+) with low attrition, and instituting Scrum ceremonies and sprint cadence — cutting lead time 25% and reducing change failure rate 35%.

Track record

When purpose leads and the data backs it, the numbers move.

A few of the decisions I've helped shape leading AI engineering and global transformation — and what changed as a result.

Manual processing time
−65%
LLM document intelligence, enabling 3× higher document volumes with no added headcount.
Model deployment cycles
weeks → days
Dual-cloud platforms on AWS & Azure, with ML infrastructure costs cut by 30%.
Production AI incidents
−45%
Runtime guardrails and real-time observability; drift detection from days to hours.
+35% velocitysprint velocity, with NPS up 18%
20+ teamcross-functional AI/ML org, <5% attrition
+10% revenuenew AI-powered subscription models
1,200 serversmigrated to Azure, TCO −25%, 99.9% uptime
SOC 2 Type IIcompliance across production AI systems
−70% release cyclesAKS + CI/CD, 600+ engineering hours / yr saved
Where I've done it

Operating experience, not just advice.

2023 — Now

VP, AI/ML Engineering · Block 64, Toronto, ON

Defined and executed the AI-First transformation strategy while staying hands-on in architecture — building a 20+ member AI/ML organization with 4 senior leaders reporting in; and dual-cloud platforms on AWS and Azure, with revenue up 10% through new AI-powered subscription models.

Introduced GenAI-powered, agentic AI ecosystems and embedded Claude Code into delivery workflows, lifting delivery velocity 35% and NPS 18%.

AI-First Transformation · Block 64

2022 — 2023

Director, Software Engineering · Block 64, Toronto, ON

Owned a $5M cloud and AI platform portfolio across AWS and Azure, grew the team from 12 to 20+ with under 5% attrition, and introduced AI-assisted delivery practices that improved predictability 35% and reduced defects per sprint 10%.

Rearchitected infrastructure from legacy AWS IaaS/on-premises to Azure PaaS/Serverless, cutting infrastructure costs 30% while introducing SOC 2 Type II compliance and outsourcing infra maintenance to a managed services partner — freeing engineering to focus fully on product and system-architecture improvements.

2018 — 2021

Global Transformation Lead · Lexmark Canada, Markham, ON

Led delivery across a $100M+ global managed services portfolio, managing the Americas, EMEA, and APAC teams on a follow-the-sun model — with clear onshore/offshore handoff discipline and lifting client CSAT to 80% (5 to 9/10).

Implemented an IoT ML-driven predictive-maintenance service to a major Canadian bank from a phased pilot-to-mass-deployment rollout across a 15,000-device fleet — lifting SLA response time 35%, helping secure a multi-year global contract renewal.

2015 — 2018

Global Delivery Manager · Hewlett-Packard Canada, Toronto, ON

Owned delivery across a $75M+ global managed-services contract serving a major Canadian bank, spanning the Americas, Europe and the Caribbean teams.

Directed enterprise IT transformation and cloud migration — reducing data-center costs 20% and increasing system availability 25% under enterprise governance, SLA/OLA, and control standards, while leading cross-functional teams across operations, procurement, and billing.

2011 — 2015

Technology & Solutions Architect · Hewlett-Packard Canada, Toronto, ON

Delivered SOW-based digital workflow transformations for financial-services clients, operating within regulated financial-services governance and internal-control frameworks — compressing process cycle times from a day to 45 minutes and generating $500K in annual savings. Improved operational productivity by 20% through custom system redesign and automated data capture.

Advised enterprise clients on consultancy model architecture and digital transformation strategy — including embedded imaging/printing hardware and firmware for banking, insurance, legal, healthcare, and government workflows — building familiarity with regulated delivery models and internal control standards.

2009 — 2010

Business Development Manager · EcoCarrier Telecom, Markham, ON

Created successful business development, sales, and marketing activities for the Telecom industry, offering a unique managed services portfolio focused on the international wholesale market for voice, text, and data roaming services — resulting in a 10% increase in market share YoY.

2000 — 2008

Pre-Sales Engineer & Consultant · Lexmark Brazil & HP Brazil, São Paulo, SP

Drove both go-to-market motions: direct pre-sales consulting (RFP/RFQ responses, POCs, Sales Pursuit Teams) into banking, insurance, legal, healthcare, and government accounts; and partner-channel enablement focused on SMB — training the partner network on the solution portfolio and running channel-side pilots and POCs.

Projects

Things I've built, not just led.

Hands-on builds — agentic systems, evaluation harnesses, cloud-native architecture. Code is public; each links straight to the repo.

Real-Time Video Surveillance on Azure AI Vision

A cost-aware, end-to-end video surveillance and alerting system — browser or IP camera capture, per-frame Vision analysis, live WebSocket alerts plus email/SMS. Built solo in two weeks, deployed via a single resumable CLI with zero portal clicks.

  • Azure AI Vision
  • Azure Functions
  • Container Apps
  • FastAPI
  • React/TypeScript
  • Bicep

Serverless-first architecture where only storage and the Vision resource run continuously — compute scales to zero. Pluggable analyzer interface allows a Custom Vision or YOLO detector swap without touching capture, alerting, or deployment. Root-caused a Google SDM API limitation on WebRTC video against real multi-camera hardware.

View on GitHub →

Monolith → Microservices: A Live Strangler Fig Cutover

The same e-commerce application running two ways — one Flask monolith on a shared database, and three independently deployable services each owning its own. Trigger the migration and watch a real cutover happen, not an animation.

  • Python
  • Flask
  • FastAPI
  • React/TypeScript
  • Azure Container Apps
  • BFF proxy routing
  • Anti-Corruption Layer

Proxy-based traffic shifting with shadow testing and staged live-percentage cutover. Identical behaviour verified on both architectures by an automated smoke suite. Cross-service auth implemented as a real HTTP boundary rather than a shared database read.

View on GitHub →

Agent Orchestration: 10-Protocol Benchmark

A production-grade multi-agent customer support system, then a head-to-head benchmark of ten inter-service communication protocols — each wired into a real specialist agent's tool-calling loop rather than a synthetic microbenchmark.

  • LangGraph
  • MCP
  • Google A2A
  • REST
  • GraphQL
  • gRPC
  • WebSocket
  • MQTT
  • AMQP
  • SOAP
  • Webhook
  • FAISS
  • LangSmith
  • Langfuse

Supervisor routing with deterministic quick-answer bypass for roughly a third of tickets — no LLM call where none is warranted. Human-in-the-loop escalation via graph interrupts, PII redaction before model exposure, cross-session memory. Shared fault-injection and timing seam makes all ten protocols directly comparable.

View on GitHub →

Eval Harness: 7 RAG Patterns, One Scoring Framework

Seven RAG architectures benchmarked against the same 40-question dataset and corpus, scored by a common harness — then wired into CI so quality regressions fail the build automatically.

  • LangChain/LangGraph
  • FAISS
  • ChromaDB
  • RAGAS
  • LLM-as-judge
  • GitHub Actions

Retrieval metrics (Hit Rate, MRR, nDCG) plus generation metrics (groundedness, hallucination rate) and RAGAS scores across every pattern. Regression check exits non-zero when any metric drops beyond threshold or hallucination rate climbs — the guardrail that lets a fast-moving team ship without silently degrading.

View on GitHub →

Five Agent Frameworks, One Compliance Pipeline

The same regulatory reporting pipeline implemented five times — LangGraph, OpenAI Agents SDK, CrewAI, AutoGen, and Google ADK — then compared on architecture, timing, and orchestration model.

  • LangGraph
  • OpenAI Agents SDK
  • CrewAI
  • AutoGen
  • Google ADK
  • LangSmith

Parallel specialist fan-out with an evaluator loop that iterates each section until numerical accuracy, terminology, and cross-reference checks pass. Human review gate before final output. Side-by-side runtime and LLM-call comparison across all five — a framework-selection decision grounded in measurement rather than preference.

View on GitHub →

AWS Bedrock AgentCore: Memory, RAG and a Secured Gateway

An autonomous support agent on Bedrock AgentCore with cross-session memory retention, contextual retrieval, and a JWT-secured tool gateway — fully provisioned as code.

  • AWS Bedrock AgentCore
  • Bedrock Knowledge Bases
  • MCP
  • OpenTelemetry
  • CloudFormation

Full telemetry instrumentation across the agent lifecycle. Architecture projects a 30–50% structural compute cost reduction versus always-on inference hosting.

View on GitHub →
All repositories →
Credentials

Qualified, and current.

Qualifications

MBA, International BusinessRotman School of Management & Business School São Paulo (BSP)
B.Sc. Computer ScienceUniversidade Ibirapuera, Brazil

Certifications & Executive Education

PMP & SAFe AgilistCSP-ScrumMaster
ITIL v4 Foundation & Prosci/ADKARService management & change management
MIT Executive EducationNo-Code AI and ML · Digital Transformation

Cloud & AI/ML Certifications

AWS Solutions Architect AssociateSAA-C02
Microsoft Azure AI Engineer AssociateAI-102
Microsoft Azure AI FundamentalsAI-901
Microsoft Azure FundamentalsAZ-900
Microsoft AI Transformational LeaderAB-731

Tools & systems

Cloud Platforms — AWS (Bedrock, SageMaker, Lambda, EventBridge, Step Functions, CloudWatch), Azure (Microsoft Foundry, Azure AI Search, AKS, Container Apps, Functions, Logic Apps).
GenAI & LLM Engineering — LangChain, LangGraph, MCP, Google A2A, CrewAI, AutoGen, RAG, Multi-Agent Orchestration, Claude Code CLI, OpenAI Codex API (Automated PR Reviews), Pydantic (Structured Outputs), Anthropic Claude, OpenAI GPT, Azure OpenAI.
Data Science & ML — Python (Pandas, NumPy, Scikit-learn, TensorFlow, PyTorch), Statistical Modelling, A/B Testing, Feature Stores.
Data & Integration — PostgreSQL, SQL & NoSQL (DynamoDB), Vector Databases (Pinecone, ChromaDB, FAISS), ETL/ELT Pipelines, REST, GraphQL, gRPC.
CI/CD & Pipeline Infrastructure — GitHub Actions, Azure DevOps, Jenkins, Octopus Deploy, Docker, Kubernetes (AKS), Terraform, Bicep, CloudFormation, DORA Metrics Alignment.
Progressive Delivery & Feature Control — OpenFeature Architecture, Octopus Deploy Flags SDK, Runtime Targeting Gates & Circuit Breakers.
Observability & Reliability — OpenTelemetry, LangSmith, Langfuse, Elasticsearch, Application Insights, Azure Monitor, CloudWatch, SLO/SLA Instrumentation.
Testing & Quality Automation — pytest, Vitest, LLM Evals (RAGAS, LLM-as-Judge), CI Regression Gates.
AI/ML Platform & MLOps — MLflow (Tracking, Model Registry), SageMaker Endpoints, Azure ML Endpoints, Model & Data Drift Detection, Experiment Tracking & Model Versioning.
Governance & Compliance Frameworks — SOC 2 Type II (Drata), NIST AI RMF, Credo AI, ISO 27001 & ISO 42001 (in progress, target 2028), Responsible AI Governance, SLA/OLA Management, ITIL v4.

Get in touch

Tell me the outcome you need AI to deliver.

An AI strategy, a production platform, an agentic system, or an AI/ML team that needs leadership — start with a short call and we'll work out whether I'm the right fit.

Email me → Connect on LinkedIn
Emailmarcosoliveira.connect@gmail.com
LinkedInlinkedin.com/in/mfilho1
GitHubgithub.com/MAOFILHO
Based inGreater Toronto Area