OpenCV MNKY: Building a Gesture-Controlled Avatar System

Using Python, OpenCV, and MediaPipe to create a real-time gesture recognition system.

The Idea

I wanted to build something fun that combined computer vision with interactive media. The result: OpenCV MNKY — a gesture-controlled avatar meme system that changes images based on your hand gestures in real-time.

How It Works

The system uses three main technologies:

1. OpenCV for camera capture and image processing
2. MediaPipe for face and hand detection
3. NumPy for efficient array operations

The Detection Pipeline

import cv2
import mediapipe as mp

mp_face = mp.solutions.face_detection
mp_hands = mp.solutions.hands

face_detection = mp_face.FaceDetection(
    model_selection=0, 
    min_detection_confidence=0.6
)

hands = mp_hands.Hands(
    max_num_hands=2, 
    min_detection_confidence=0.6
)

Gesture States

The avatar changes based on what's detected:

| Detection | Avatar | Meaning | |-----------|--------|---------| | Face only | mon3.jpg | Neutral state | | Face + Hand | mon2.jpg | Active/waving | | Finger near mouth | mon1.jpg | "Shh" gesture | | No detection | Black screen | Nobody home |

Technical Challenges

Challenge 1: Camera Selection

Laptop webcams are usually index 0, but external cameras (Camo, OBS Virtual Camera) have different indices. I implemented auto-detection:

for cam_index in range(1, 6):
    cap = cv2.VideoCapture(cam_index)
    if cap.isOpened():
        print(f"✓ Camera index {cam_index} found!")
        break

Challenge 2: Mirror Effect

People expect to see themselves mirrored (like a real mirror). The solution:

frame = cv2.flip(frame, 1)  # Horizontal flip

Challenge 3: Detecting "Finger Near Mouth"

This required calculating the distance between finger landmarks and face landmarks:

# Simplified logic
mouth_y = face_landmarks[0].y * frame_height
finger_y = hand_landmarks[8].y * frame_height  # Index finger tip

if abs(mouth_y - finger_y) < threshold:
    show_shh_avatar()

Performance Optimization

Real-time computer vision is demanding. Optimizations I made:

1. Reduce resolution — 720p is enough for detection
2. Skip frames — Process every 2nd frame if needed
3. ROI processing — Only analyze relevant regions
4. GPU acceleration — Use MediaPipe's GPU inference when available

The Split-Screen Layout

The final output shows:

• Left side: Your mirrored camera feed
• Right side: Avatar that changes with gestures

combined = np.hstack([frame_resized, avatar_resized])
cv2.imshow('OpenCV MNKY', combined)

What I Learned

1. MediaPipe is incredible — Google's ML solutions work out of the box
2. Real-time is hard — 30 FPS means you have ~33ms per frame
3. User experience matters — Fullscreen toggle and smooth transitions make a difference

Future Improvements

Ideas for version 2.0:

• Multiple gesture recognition (peace sign, thumbs up)
• Custom avatar upload
• OBS integration for streaming
• Facial expression detection

---

Repository: github.com/Airyshtoteles/OpenCV_MNKY

MIT Licensed — feel free to fork and create your own gesture-controlled memes!