swscale/output: Fix integer overflow in yuv2gbrp_full_X_c()

Fixes: signed integer overflow: 1966895953 + 210305024 cannot be represented in type 'int'
Fixes: 391921975/clusterfuzz-testcase-minimized-ffmpeg_SWS_fuzzer-5916798905548800

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ce538ef97a7b1fdab6f2a3c8afc538c1cc3760d9)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This commit is contained in:
Michael Niedermayer
2025-02-04 03:58:44 +01:00
parent eb88a2318e
commit b3f9eac35a

View File

@ -2199,9 +2199,9 @@ yuv2gbrp_full_X_c(SwsContext *c, const int16_t *lumFilter,
Y -= c->yuv2rgb_y_offset;
Y *= c->yuv2rgb_y_coeff;
Y += 1 << (SH-1);
R = Y + V * c->yuv2rgb_v2r_coeff;
G = Y + V * c->yuv2rgb_v2g_coeff + U * c->yuv2rgb_u2g_coeff;
B = Y + U * c->yuv2rgb_u2b_coeff;
R = Y + V * (unsigned)c->yuv2rgb_v2r_coeff;
G = Y + V * (unsigned)c->yuv2rgb_v2g_coeff + U * (unsigned)c->yuv2rgb_u2g_coeff;
B = Y + U * (unsigned)c->yuv2rgb_u2b_coeff;
if ((R | G | B) & 0xC0000000) {
R = av_clip_uintp2(R, 30);