This content is created in collaboration with the Glossic team, they sell some of the awesome lip balm for dry lips, as we designed this Python script for them and they were fine with us posting the full thing here on FSI Blog.
I see that the virtual lipstick try on thing is on every second beauty site now, you upload a selfie, tap a shade, and your lips are that colour. And everyone I talk to assumes it is ChatGPT or Gemini running behind it. No. These sites were doing it in 2018 and 2019, L’Oreal bought ModiFace in 2018 just for this, Sephora had it in the app even before, and there was no ChatGPT then, no Gemini, no Grok live face API, nothing. So it is a library. And if it is a library there is a pip command for it, right!
As a developer and a bit of an SEO geek I wanted to know how the logic works, so here it is, plus the script.
Now I am not here to prove which exact SDK the big brands are running, that is not the agenda. ModiFace has a bigger team than most agencies. But the pipeline is the same everywhere:
- Find the face.
- Find the lips inside the face.
- Make a mask of the lips.
- Blend a colour into the mask.
Four steps. All four open source. All four older than the AI boom.
What Is The Library Then?
Three options, and I have used all three on client work.
OpenCV, pip install opencv-python
- Oldest one, around since the year 2000.
- Ships with Haar cascade face detection, no download.
- Gives you a rectangle around the face and that is it.
- No idea where the mouth is inside that rectangle, so on its own it is useless for lips.
- This is what most of the 2018 to 2020 try on sites were built on, or something that looks like it.
- 68 point face landmarks, and points 48 to 67 are the mouth, outer lip 48 to 59, inner lip 60 to 67.
- Needs CMake and a C++ compiler to install, on Windows it fails for half the people, I have watched it happen on client calls.
- Plus a separate 99 MB model file
shape_predictor_68_face_landmarks.datyou download yourself. - Works, just a bad afternoon.
MediaPipe (Google, 2019), pip install mediapipe
- No compiler, no model file, the model is inside the wheel.
- 468 points on the face instead of 68, about 40 of them are lips, so the outline is smooth.
- Runs one image in under a second on a normal laptop CPU.
- This is the one the script uses.
So the framework thing is simple here, go with MediaPipe unless you have some reason to suffer.
Instant answer for the “is it real time AI” question then, NO. A landmark model is machine learning, yes, but it is 2019 machine learning and it runs on the user’s own device without calling anyone’s API. Nobody is burning the $ here.
Which Points Are The Lips?
MediaPipe gives you 468 numbered points and nowhere does it say “this one is lip”. You have to know the numbers. This took me a while the first time so I am writing them down:
- Outer lip, going round: 61, 146, 91, 181, 84, 17, 314, 405, 321, 375, 291, 409, 270, 269, 267, 0, 37, 39, 40, 185.
- Inner lip, the mouth opening: 78, 95, 88, 178, 87, 14, 317, 402, 318, 324, 308, 415, 310, 311, 312, 13, 82, 81, 80, 191.
Fill the outer polygon white, fill the inner polygon black on top. What is left is a ring, the lips, teeth cut out. Skip the inner cut and the colour lands on the teeth when the mouth is open. I have seen a live site doing exactly that.
How The Colour Goes On Without Looking Like Paint
If you just paint a flat colour into the mask you get a sticker. No texture, no shine, no lines. Cheap try ons look like this.
The decent ones keep the brightness of the real lips and only swap the colour, and in OpenCV that means LAB colour space:
- L is lightness, A and B are the colour.
- Leave L alone, blend your shade into A and B only.
- Cracks, shine, cupid’s bow, all of that lives in L, so it stays.
- Blur the mask edge a bit so it is not cut with scissors.
Now the Glossic brief was a bit different from the lipstick clients, because a balm is barely a colour product. On a dry mouth the vertical lines and flaking sit in the L channel, so a pure colour overlay keeps them, honest but not flattering. A balm smooths those, physically. So for the balm shades the script also softens L inside the mask a little and lifts it, the software version of what a <a href=”https://www.glossic.com/product-category/lips/balmstick/” rel=”noreferrer”>lip balm for dry lips</a> does on an actual mouth. One parameter, smooth, up for balm, zero for matte.
The Script
Python 3.9 to 3.11. MediaPipe is pinned because the newer 0.10 releases moved the API around and the solutions interface below is what every tutorial on the internet still uses.
pip install opencv-python numpy "mediapipe>=0.10.0,<0.10.15"import sys
import cv2
import numpy as np
import mediapipe as mp
OUTER = [61, 146, 91, 181, 84, 17, 314, 405, 321, 375,
291, 409, 270, 269, 267, 0, 37, 39, 40, 185]
INNER = [78, 95, 88, 178, 87, 14, 317, 402, 318, 324,
308, 415, 310, 311, 312, 13, 82, 81, 80, 191]
def hex_to_bgr(h):
h = h.lstrip("#")
r, g, b = int(h[0:2], 16), int(h[2:4], 16), int(h[4:6], 16)
return (b, g, r)
def lip_mask(img, landmarks):
h, w = img.shape[:2]
pts = lambda idx: np.array(
[(int(landmarks[i].x * w), int(landmarks[i].y * h)) for i in idx],
dtype=np.int32)
mask = np.zeros((h, w), dtype=np.uint8)
cv2.fillPoly(mask, [pts(OUTER)], 255)
cv2.fillPoly(mask, [pts(INNER)], 0) # cut the mouth opening out
mask = cv2.GaussianBlur(mask, (7, 7), 0) # soft edge
return mask
def apply_colour(img, mask, bgr, alpha=0.6, smooth=0.0):
lab = cv2.cvtColor(img, cv2.COLOR_BGR2LAB).astype(np.float32)
colour_lab = cv2.cvtColor(
np.uint8([[bgr]]), cv2.COLOR_BGR2LAB)[0][0].astype(np.float32)
m = (mask.astype(np.float32) / 255.0)[..., None]
# keep lightness, blend only the colour channels
tinted = lab.copy()
tinted[..., 1] = lab[..., 1] * (1 - alpha) + colour_lab[1] * alpha
tinted[..., 2] = lab[..., 2] * (1 - alpha) + colour_lab[2] * alpha
if smooth > 0:
# balm mode: soften the lines in the lightness channel a little
L = lab[..., 0]
L_soft = cv2.bilateralFilter(L, 9, 20, 9)
tinted[..., 0] = L * (1 - smooth) + L_soft * smooth + smooth * 6
out = lab * (1 - m) + tinted * m
out = np.clip(out, 0, 255).astype(np.uint8)
return cv2.cvtColor(out, cv2.COLOR_LAB2BGR)
def main():
if len(sys.argv) < 3:
print("usage: python lips.py input.jpg '#B03A48' [alpha] [smooth]")
sys.exit(1)
path = sys.argv[1]
bgr = hex_to_bgr(sys.argv[2])
alpha = float(sys.argv[3]) if len(sys.argv) > 3 else 0.6
smooth = float(sys.argv[4]) if len(sys.argv) > 4 else 0.0
img = cv2.imread(path)
if img is None:
print("could not read", path)
sys.exit(1)
mesh = mp.solutions.face_mesh.FaceMesh(
static_image_mode=True, max_num_faces=1, refine_landmarks=True)
res = mesh.process(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
if not res.multi_face_landmarks:
print("no face found")
sys.exit(1)
lm = res.multi_face_landmarks[0].landmark
mask = lip_mask(img, lm)
out = apply_colour(img, mask, bgr, alpha, smooth)
cv2.imwrite("out.jpg", out)
print("saved out.jpg")
if __name__ == "__main__":
main()
Run it. Matte red:
python lips.py selfie.jpg "#B03A48"Sheer balm, low colour, high smoothing:
python lips.py selfie.jpg "#C8707A" 0.3 0.6I ran it on my own LinkedIn photo first, same as I did for the buzz cut post. Am I looking nice guys?
Things that went wrong for us, so they will go wrong for you:
- Photo wider than about 1500 px, the 7×7 blur on the mask turns into a hard edge, resize down first or scale the kernel.
- Face turned more than about 40 degrees, the inner lip points cross the outer ones and the mask gets a hole in the wrong place.
refine_landmarks=Truematters, without it the lip points are rough, I do not know why the default is False.
If You Are The One Selling The Lipstick
So the logic is, you do not need an API bill. The RightHair type sites charging per generation because they call Nano Banana behind the scenes, that model makes sense when the output is a brand new image. Lips are not that. Lips are a known shape on a known face and a 2019 library finds them for free on the user’s own phone.
Two ways we have shipped this for clients:
- A small Flask or FastAPI endpoint wrapping the script, upload in, JPEG out.
- The whole thing in the browser with MediaPipe’s JavaScript build, so the photo never leaves the phone, which the privacy people like.
Our Python side did the first version of the Glossic one, the React side is doing the browser port. Sixty odd developers in the building means those two things happen in the same week, which is honestly the only reason I get time to write posts instead of debugging CMake for dlib on somebody’s Windows laptop.
Try it on your own face first, then on your product colours. If a shade looks wrong in the script it usually looks wrong on the product page too, and that has saved more than one client a reshoot.