FRR MP-BGP route learning issue in Kernel

[Madhu-api]

I have made an FRR MP-BGP setup referencing
https://github.com/martimy/clab_mpls_frr/blob/main/img/mpls_vrf.png

(setup diagram attached)
All are ubuntu 20.04 FRR docker containers

madhuQAVM1# show vers
FRRouting 10.5.0-my-manual-build (madhuQAVM1) on Linux(5.4.0-193-generic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
'--prefix=/usr' '--localstatedir=/var/run/frr' '--sbindir=/usr/lib/frr' '--sysconfdir=/etc/frr' '--enable-sharpd' '--enable-multipath=64' '--enable-user=frr' '--enable-group=frr' '--enable-config-rollbacks' '--enable-vty-group=frrvty' '--enable-snmp=agentx' '--enable-scripting' '--with-pkg-extra-version=-my-manual-build'

Topology

R4 and R5 are CE routers in vrf blue with default route configured towards R1 and R3 respectively
R6 and R7 are CE routers in vrf red with default route configured towards R1 and R3 respectively

R1 and R3 have vrf blue and red configured. R2 is P-core router and doesn't have any VRF config

Problem statement:

unable to ping end-to-end, i.e,
unable to ping R5 from R4 in vrf blue
unable to ping R7 from R6 in vrf red

in PE routers R1 and R3, the end routes are actually getting learned in FRR
192.168.71.0/24 is the concerned route here

FRR command outputs

madhuQAVM1# show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 1.1.1.1, local AS number 100 VRF default vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 24 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
3.3.3.3         4        100      4633      4636        0    0    0 3d05h03m            0        0 FRRouting/10.5.0-my-

Total number of neighbors 1

IPv4 VPN Summary:
BGP router identifier 1.1.1.1, local AS number 100 VRF default vrf-id 0
BGP table version 0
RIB entries 3, using 384 bytes of memory
Peers 1, using 24 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
3.3.3.3         4        100      4633      4636       16    0    0 3d05h03m            4        4 FRRouting/10.5.0-my-

Total number of neighbors 1

madhuQAVM1# show ip bgp vrf blue
BGP table version is 10, local router ID is 192.168.65.219, vrf id 7
Default local pref 100, local AS 100
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.55.0/24  3.3.3.3@0<               0    100      0 ?
 *>  192.168.65.0/24  0.0.0.0                  0         32768 ?
 *>  192.168.70.0/24  192.168.65.229           0         32768 ?
 ***>  192.168.71.0/24  3.3.3.3@0<               0    100      0 ?**

Displayed 4 routes and 4 total paths

madhuQAVM1# show ip route vrf blue
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

IPv4 unicast VRF blue:
B>  192.168.55.0/24 [200/0] via 3.3.3.3 (vrf default) (recursive), label 80, weight 1, 3d02h42m
  *                           via 192.168.64.243, enp0s9 (vrf default), label 16/80, weight 1, 3d02h42m
C>* 192.168.65.0/24 is directly connected, enp0s10, weight 1, 3d03h58m
L * 192.168.65.219/32 is directly connected, enp0s10, weight 1, 3d03h58m
L * 192.168.65.219/32 is directly connected, enp0s10, weight 1, 3d03h58m
L>* 192.168.65.219/32 is directly connected, enp0s10, weight 1, 3d04h29m
S>* 192.168.70.0/24 [1/0] via 192.168.65.229, enp0s10, weight 1, 3d04h29m
**B>  192.168.71.0/24 [200/0] via 3.3.3.3 (vrf default) (recursive), label 80, weight 1, 3d02h42m
  *                           via 192.168.64.243, enp0s9 (vrf default), label 16/80, weight 1, 3d02h42m**

but, when we check the same routes in kernel it is missing

root@madhuQAVM1:/go/src/app# ip vrf exec blue ip route show

2.2.2.2 nhid 36 via 192.168.64.243 dev enp0s9 proto ospf metric 20
3.3.3.3 nhid 37  encap mpls  16 via 192.168.64.243 dev enp0s9 proto ospf metric 20
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.61.0/24 nhid 36 via 192.168.64.243 dev enp0s9 proto ospf metric 20
192.168.64.0/24 dev enp0s9 proto kernel scope link src 192.168.64.229

no 192.168.71.0/24 route here

However, following commands are showing the expected routes

root@madhuQAVM1:/go/src/app# ip route show vrf blue
192.168.55.0/24 nhid 41  encap mpls  16/80 via 192.168.64.243 dev enp0s9 proto bgp metric 20
192.168.65.0/24 dev enp0s10 proto kernel scope link src 192.168.65.219
192.168.70.0/24 nhid 26 via 192.168.65.229 dev enp0s10 proto 196 metric 20
192.168.71.0/24 nhid 41  encap mpls  16/80 via 192.168.64.243 dev enp0s9 proto bgp metric 20

root@madhuQAVM1:/go/src/app# ip route show table 1
192.168.55.0/24 nhid 41  encap mpls  16/80 via 192.168.64.243 dev enp0s9 proto bgp metric 20
broadcast 192.168.65.0 dev enp0s10 proto kernel scope link src 192.168.65.219
192.168.65.0/24 dev enp0s10 proto kernel scope link src 192.168.65.219
local 192.168.65.219 dev enp0s10 proto kernel scope host src 192.168.65.219
broadcast 192.168.65.255 dev enp0s10 proto kernel scope link src 192.168.65.219
192.168.70.0/24 nhid 26 via 192.168.65.229 dev enp0s10 proto 196 metric 20
192.168.71.0/24 nhid 41  encap mpls  16/80 via 192.168.64.243 dev enp0s9 proto bgp metric 20

root@madhuQAVM1:/go/src/app# ip vrf exec blue ping 192.168.71.181
ping: connect: Network is unreachable
root@madhuQAVM1:/go/src/app#

host ping:

root@madhuQAVM4:/go/src/app# ping 192.168.71.181 -c 5
PING 192.168.71.181 (192.168.71.181) 56(84) bytes of data.

--- 192.168.71.181 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4233ms

Please let me know how to make the configuration work in the scenario
Full configuration in all routers attached

note:
vrf was configured using the following cmds in PE routers

sudo ip link add blue type vrf table 1
sudo ip link add red type vrf table 2
ip addr flush dev enp0s8
ip addr flush dev enp0s10
sudo ip link set enp0s10 vrf blue
sudo ip link set enp0s8 vrf red
ip addr add 192.168.62.111/24 dev enp0s8
ip addr add 192.168.65.219/24 dev enp0s10
sudo ip link set blue up
sudo ip link set red up

Full FRR config:
mpls MP-BGP config.txt

Let me know if you need any more outputs, will attach it

[donaldsharp]

When you are using l3mvrf devices they are not associated with a namespace at all. They are just vrf's in the traditional sense. I do not understand why you are doing exec commands inside of a namespace to gather data since that is not what is configure from what I can tell.

[Madhu-api]

When you are using l3mvrf devices they are not associated with a namespace at all. They are just vrf's in the traditional sense. I do not understand why you are doing exec commands inside of a namespace to gather data since that is not what is configure from what I can tell.

Thanks. vrf configuration is done in 'PE' routers and I'm trying to initiate end-to-end ping from the host 'CE' devices, which doesn't have any vrf. the configuration is done as per the reference given in the above link. pls let me know if any configuration I'm missing in PE devices w.r.t VRF. I was checking routes in PE since the end-to-end traffic was getting dropped in the PE
(R1)