To filter a protocol encapsulated into TCP/UDP you should check EthType field for ETHERTYPE_IP = 0x0800, then check if Protocol field value is equal to 6 or 17 in the IP packet, and then check SrcPort/DstPort fields values in the TCP/UDP packet.

SMTP packets filter example:

IPPROTO_ICMP      = 1  //  ICMP protocol
IPPROTO_TCP       = 6  //  TCP  protocol
IPPROTO_UDP       = 17 //  UDP  protocol

See also files winsock.h/winsock2.h (Microsoft Platform SDK).

BpfAddCmd(hFtr,BPF_LD+BPF_H+BPF_ABS, 12);                    // Get EthType field value
BpfAddJmp(hFtr,BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_IP, 0, 10);  // EthType == ETHERTYPE_IP
BpfAddCmd(hFtr,BPF_LD+BPF_B+BPF_ABS, 23);                    // Get Protocol field value
BpfAddJmp(hFtr,BPF_JMP+BPF_JEQ+BPF_K, IPPROTO_TCP, 0, 8);    // IP_HEADER.Protocol == IPPROTO_TCP
BpfAddCmd(hFtr,BPF_LD+BPF_H+BPF_ABS, 20);                    // Get Flags value
BpfAddJmp(hFtr,BPF_JMP+BPF_JSET+BPF_K, 0x1fff, 6, 0);        // Flags & 0x1fff (reject fragmented packets)
BpfAddCmd(hFtr,BPF_LDX+BPF_B+BPF_MSH, 14);                   // X <- 4*(Len_Ver & 0xf) 
BpfAddCmd(hFtr,BPF_LD+BPF_H+BPF_IND, 14);                    // Get SrcPort field value
BpfAddJmp(hFtr,BPF_JMP+BPF_JEQ+BPF_K, 25, 2, 0);             // SrcPort == 25
BpfAddCmd(hFtr,BPF_LD+BPF_H+BPF_IND, 16);                    // Get DstPort field value
BpfAddJmp(hFtr,BPF_JMP+BPF_JEQ+BPF_K, 25, 0, 1);             // DstPort == 25
BpfAddCmd(hFtr,BPF_RET+BPF_K, (UINT)-1);                     // TRUE  (Accept)
BpfAddCmd(hFtr,BPF_RET+BPF_K, 0);                            // FALSE (Reject)

#define IP_PROTO   0x800  
#define TCP_PROTO  6  
#define SMTP_PORT  25  
     ld   P[12:2]                     // A = WORD offset 12 (protocol in the Ethernet header)
     jeq  IP_PROTO, 0, Exit           // If A <> 0x800 (IP), exit
     ld   P[23]                       // A = BYTE offset 23 (protocol in the IP header)
     jeq  TCP_PROTO, 0, Exit          // If A <> 6 (TCP), exit
     ld   P[20:2]                     // A = WORD offset 20 (IP flags)
     jset 0x1FFF, Exit, 0             // A & 0x1FFF - reject fragmented packets
     ldxm P[14]                       // X = length of the IP header = 4*(P[14] & 0x0F)
     ld   P[X+14:2]                   // A = Source port
     jeq  SMTP_PORT, OK_ThisIsSMTP, 0 // If A = SMTP_PORT (25), OK
     ld   P[X+16:2]                   // A = Destination port
     jeq  SMTP_PORT, 0, Exit          // If A <> SMTP_PORT (25), then it is not an SMTP packet
OK_ThisIsSMTP:
     ret  -1                          // It is SMTP packet, exit and return TRUE
Exit:
     ret  0                           // It is not an SMTP packet, exit and return FALSE

To learn more about port numbers please refer to http://www.iana.org/assignments/port-numbers.

TCP (Transmission Control Protocol, RFC-793, RFC-1323) packet structure:

SrcPort

Source port (16 bits)

DstPort

Destination port (16 bits)

Sequence Number

Sequence Number (32 bits). The sequence number of the first data byte in the packet. Unless the SYN flag is set in which case the sequence number is the ISN (initial sequence number) and the first data byte has a sequence number of ISN + 1.

Acknowledgment Number

Acknowledgment Number (32 bits). If the ACK bit is set, this number is the next sequence number the sender of the packet expects to receive. After a connection is established, the ACK bit will always be set and this number will always be used.

Data Offset

Data Offset (4 bits). The number of 32 bit words in the TCP header which will probably be five (5) unless you use Options.

Reserved

Reserved (6 bits). Reserved for future use.

Control Bits

Control Bits (6 bits).

Window

Receive window size (16 bits).

Checksum

Checksum (16 bit).

UDP (User Datagram Protocol, RFC-768) packet structure:

SrcPort

Source port

DstPort

Destination port

Length

Datagram length in octets

CheckSum

Check sum