Video: DEMO: Troubleshooting FTP Issue (Blocking Activities, Creating Policies) | Duration: 941s | Summary: Resolve FTP breaking issue by creating custom app and cloud/web access policies for client update problems. Video: Inside Netskope: Identify, Troubleshoot, and Resolve Common Netskope Client-related Issues | Duration: 3672s | Summary: Inside Netskope: Identify, Troubleshoot, and Resolve Common Netskope Client-related Issues | Chapters: Introduction to Troubleshooting (2.32s), Netskope Client Architecture (367.52s), Dynamic Steering Configuration (652.885s), Secure Enrollment Troubleshooting (926.51s), Advanced Log Analysis (1134.135s), Implementing Web Policies (1992.005s), Q&A and Troubleshooting (2307.125s), Reading Client Logs (2579.22s), Resources for Admins (2695.075s), Troubleshooting Client Issues (2802.5498s), Network Latency Troubleshooting (2898.175s), Log Analysis Techniques (3025.57s), Troubleshooting Netskope Issues (3163.97s), POP Switching Limitations (3414.8s), Alert Action Availability (3455.875s), Log Analysis Techniques (3501.3801s), Troubleshooting and Conclusion (3545.425s)
Transcript for "Inside Netskope: Identify, Troubleshoot, and Resolve Common Netskope Client-related Issues": Hey. Good morning, folks. We're gonna give, another thirty seconds or so for everybody to get a chance to get joined up, grab that last cup of coffee, etcetera. So just hold out for a minute. Enjoying the awkward silence? Alright. Let's get rolling here. Today on Inside Netskope here, we're talking about identifying, troubleshooting, and resolving common Netskope client related issues, because any security tool out there just, you know, interferes with other things on your systems. And sometimes you just gotta be able to figure out how to correct that and, you know, get things going smoothly for your for your end user center. So today, you'll be, hearing from myself, Robert Butler. I'm part of the customer zero team and the security operations center. And my other counterpart here, Stevan Pierce, master of magician activities, within all things Netskope. And, yeah, so without further ado, let's, let's get it rolling. Few, housekeeping items. Slides, from this presentation will be available upon request, at least twenty four hours after, we get going here. The recording will be available as well. It'll be posted in our our, webinar hub, and it could be a link, via the email also that will have a follow-up email that comes out. Q and a is available during the event. It'll be on the tab on your right within Goldcast here. And there is a poll with potential future topics that'll be shared during the event. So please make sure, you know, you vote it's it's our version of you choose the news. Right? So, you jump in there, and, these are topics that that we're excited about, we're passionate about, things that we have come across that have been helpful for us. And, you pick which one you wanna hear from, and we'll get the right people in to, you know, talk talk with you guys about, like, hey. This has been our experience and whatnot. So wanted to, just to let everybody know if you haven't seen it yet, we have a brand new customer portal that's been rolled out there. Much better search capabilities, a lot more information all in one spot. The Netskope intelligent assistant is, has been, really helpful. I know I've done a few things myself where, like, I'm trying to find stuff to reference for some encryption for an audit, and it's like, I don't wanna dig for that information. It even not only gave me the answer, but told me where to find it in the documentation to provide that for audit items. So it is a pretty cool tool, to have in your pocket there. And, just like everything else, that we're trying to do here with our our webinars, we usually have a corresponding, write up about what we've done and and what it's been, good for us with, and the issues that we've had also, some of the gotchas, we always try to point those out too, is in the Netskope community. There's a lot of good information in there, not just about what we have going on. We have other, SEs and a lot of customer interaction, within the whole thing. There's a lot of good content that can help you, just run things smoother and get get more, bang for your buck out of the toolset. So please jump in there. If you comment on one of our articles, our teams are trying to watch those, make sure we're responding and getting the answers that you're looking for. So just with, you know, this is it. We are customer zero. We run our own product. We run Netskope at Netskope to protect Netskope from the business interest side. So we're out there in the trenches doing doing this just like you guys are, but we're using, you know, our own product to do it. Right? We wanna use all Netskope features and operationalize them, and we are continuing to build build a modernized global SOC and running improvements on that as we go, and learning how better to run use Netskope as the backbone of all that. And, and our our idea here is to lead the way by example, learn from our mistakes, and then share that with you so you don't make those same mistakes. And today's session is, the the backbone of the the whole, you know, endpoint traffic control is a client troubleshooting. Yeah. Thanks. Thanks, Robert. So when they when they asked me to actually speak on this and write an article, I was excited. And then it just dawned on me that this topic, it's pretty wide. We have an actual week plus course that our support engineers go through just to troubleshoot. Now I'll be it's not all client related problems, but that'll give you the idea of the, breadth and width of the, troubleshooting that we do in Netskope. So as customer zero, what we're gonna cover is we're not gonna cover everything. There's no way we can cover it in the time that we have. So we're gonna talk about the things that we've seen and the problems that we've experienced and just to kinda give you insights into how and and what we do for for troubleshooting. Right? So we're gonna cover, you know, Netskope client architecture because it is important to know how the client operates. Right? And the important, quote, physical components. And I say that it's I I think it's much akin to a network cable, right, on our steering and our client configurations. If those network, if the network cable's unplugged, you're not gonna get anywhere. As if your steering configuration isn't working properly, you're gonna have routing and and client performance issues. So we'll address and talk about some of the client issues we've seen and the root causes, and then we'll kinda touch on log reviews, scope of events, and and that Netskope advanced analytics. Right? And then we'll go through we'll go through a demo. So Netskope client architecture. So our Netskope client is part of our inline traffic steering mechanism. It's what we call our real time protection policies. Right? And so we perform real time protection with a client. It could be, you know, another real time steering mechanism of a IPsec or GRE tunnel, proxy chain, data plane on prem, PAC file, which is also client explicit proxy, a mobile app profiler reverse proxy. We've also added, as part of customer zero, we were able to beta test and help push out our enterprise browser, which is another form of our inline traffic steering, so another real time access method. Right? So sole purposes of what we're talking about today, again, we're just we're just digging in on the client and how we use it. So with the client, like with anything, right, could be an IPsec VPN tunnel, a GRE tunnel. What we're really doing is we're just steering traffic to the Netskope cloud and and notice I said steering traffic. So that's part of what the configuration opponents will dig into and take a look at are. So we basically create a connection from an endpoint to a data plane, and then we process the policies and rules that we've applied at which affects the behavior, going to the Internet. Right? So whether it's a browse, a download, and upload, an access for a category, an explicit application definition, cloud firewall. You know, it's all gonna go through our through our data points and affect behaviors and activities we the cloud apps or websites see. So I was talking about the the physical components, earlier, akin to, if you will, a network cable. Right? The network cable's unplugged. You're not gonna go anywhere. If the device is powered off, I can't tell you the number of times that I've actually gone through troubleshooting, only, of course, to discover, lo and behold, something was unplugged. So we'll talk about a little bit about that and and really what's in a in this traffic steering configuration. Right? So with our client configuration, we're looking at tunnel settings. Right? Whether we're doing endpoint DLP, NPA, another component of that is installing and troubleshooting as well as tamper proof. And then when we dig into the traffic steering, right, whether it's dynamic steering, if we're on prem and we need to take different paths for, borderless WAN, NPA, and so on. Right? And how we handle DNS traffic, you know, the private apps. And, again, if you're off prem or on prem, whether you're steering one or the other, all of the above. And the primary exceptions that we work with, you know, for SSL cert pinned categories, cloud firewall apps, and and application and custom applications. And in this instance, the custom applications are relevant, because we'll talk about, and we'll handle some of this in the, demo a little bit later. So some of the common client issues and root causes that we've seen, client configurations. Right? So how we process, it's top down, regardless of whether it's a client configuration or steering configuration or, for that matter, a policy in real time protection. So the first hit, it stops processing and moves on. And this is relevant because we've had users that have been long to multiple IDP groups. That's been relevant because they'll hit that first group and it stops processing. So recently, we had, somebody go to a country that we had certain restrictions in that we wanted to make sure that we had reauthentication set up for Netskope private apps as well as tamper proof, right, and failing clothes on the tunnel. As a result, you know, again, they were in a different order. They were multiple groups, and so we actually had to reorient that group, up to the top to make sure that we were fully fully protected. So similar with our steering configuration, you know, if there are multiple groups to which they belong and you're hitting that first one, that's what you're gonna see when you when you pull up your Netskope client, configuration and start to take a look at the the steering groups. Right? We also, you know, can handle what e EDLP, install and troubleshooting and and tamper proof. You know, if you take a look here, on the right hand side, the the, picture is actually where we're handling bypasses of traffic. So this is another option where we're talking to look talking about dynamic steering. Right? One of the things we tackle here are the, SSL cert pinned applications and needing to to define custom cert pinned applications. Recently, we ran into an issue where Apple decided to add a new process for their, update. And as a result, we had to go through the NetskopeDEBUG log file to take a look and figure out, okay, what is this new process doing? Where is it going? And, you know, how or what should we bypass for that? Right? I mentioned earlier on the custom application definition, and what we'll do in the demo is we're gonna define a custom app, add it to the steering configuration, then they basically, apply a real time policy, for cloud as well as web. The pictures that we have here are what it looks like when we've enabled dynamic steering. Right? On prem detection with dynamic steering. So on the left, you'll see the default tenant configuration. The left hand little button is for on prem. The one is on the right. And when you're making changes to these dynamic steering, profiles for steer excuse me, dynamic steering configuration profiles. Be forewarned that you if you wanna bypass to apply on both, you have to make that, exception. Right? I can't tell you the number of times that I've come through and duplicated things over when I'm doing, multiple dynamic steering profiles for steer just give me multiple dynamic steering config profiles. Say that three times. Right? That, you know, I've missed out on on duplication of some of those. And one of the things that we can do when we're taking a look at that configuration is, you know, if we are actually, steering traffic, whether it's on prem, again, right here, which is on the left, versus the one on the right, which is the off prem. You'll notice that in the middle picture that we have, private applications all, for both of them. So a use case that we had recently was a, an engineer that was connecting to an Azure VPN and all of our, applications, that this engineer was going to use were already in Azure. And so by default, we just said, hey. Okay. We're gonna put you on prem, by detecting, a DNS a DNS entry. Because when you VPN in, you're gonna get access to the DNS server. It's gonna resolve that DNS name. You're gonna be you're gonna be seen as on prem. And then at that point, the drop down would give us, all private apps, some of the private apps or none of the private apps. And in this case, we chose some of the private applications and then made that change, within the steering configuration so that they would be able to hit just that. And when you're taking a look at your Netskope Client configuration on the right hand side, you'll see fourth item where it says on prem check not configured and what I'm steering. Right? And so that's important to when you pull this up and you start, tackling or changing out your dynamic steering profiles to determine, you know, if this is configured. And if it is, you know, is it seeing it local, or is it seeing it off prem? So one of the frequently asked questions, that we had, for the actual webinar, and and I do wanna address it directly here, with Max. Right? So our Netskope Extension needs to be unloaded after an upgrade. And in order for us to unload that, you have to reboot the machine. So nine times out of 10, your machine is gonna work with no problem at all. But there is the rare occasion where sometimes policies not might not be followed correctly. And as a result, it's probably needing to, to be rebooted. And so if you go into a terminal session and issue the system extension control list grep Netskope Client Mac app proxy, you'll see right here where it says terminated waiting to, uninstall on reboot. So we actually have a support article on this. Windows machines are not affected because it doesn't require the the extension to be unloaded. Right? Recently, we ran into an issue with secure enrollment. So we had set up secure enrollment, on our tenant. And inadvertently, you know, the token expired. And so this is what it looks like when you're digging into an NSDEBUG log file and you see, the enrollment token error. So it actually took somebody else pointing this out to me, before I honed in on it and realized, yeah, that had expired, and we need to, to renew that. So this is what it looks like inside the, the tenant when you're doing the the secure enrollment setup. So so our advanced debugging, that's part of an option that you can configure inside, your client configuration on the install and troubleshoot tab. Down below, you'll see advanced debugging right here and then setting your log level. So another question that was asked, for this webinar is what log level do I leave it at leave it as, and when would I use debug? And my response to that is always leave it at information level, never hit debug. In the entire time that I've been working here, working on the product, I think I may have used debug level three times. One of which the support engineer actually came out and said, hey, man. We don't we don't want you using that. Just only only use it when we when we need it. And the reason behind that is is it logs a lot of data. And since we only have two log files, that are circular buffering, yes. You can imagine they get overwritten really, really quickly when, we're digging digging through and have activity on the endpoint. Right? When you're tackling and taking a look at, logs on the endpoint, what you can do is and my recommendation is to save the Windows logs Windows logs off locally and look at them. And then with the Macs, you can access those live. The reason I bring the Mac up versus the Windows is that when Windows if you have if you have that file open, the NS debugger, the NPA debugger, for that matter, any logs that are being looked at, you'll get a continual, hey. This file has changed. Do you wanna update it? Hey. This file has changed. And it's all depending upon how active that client is. With the Max, you're lucky enough that you can actually, look at look at them or review them live without necessarily having to, you know, click that okay button. Let's do it. Another thing that we use here, with our support engineers are packet captures, inners and outers. So really kind of a our tool chest, if you will, when we're supporting and submitting, issues to our support team is we take a HTTP archive, a video, sometimes a screenshot of the errors that we're seeing as well as a packet capture, inner and outer. While this might seem like a lot of information, it's much easier to provide it ahead of time than it is for that for them to come back and ask you and, you know, continue to play the, the support, the support, game back and forth. So let's take a look at some of the logs and some of the scope at events, and advanced analytics that we're dealing with. K? So the two main logs that I use are the NS Debug log as well as the, the old version. Right? And so in this example, if you'll take a look on the right hand side, for the application Google Drive, we can go into the NSDEBUG log and look for that specific file or that specific process. Right? Right now, we're seeing bypass manager, Google Drive. It's going to, you know, the play.googleapi.com. So going back to the issue that we saw with Apple adding a new process that needed to be bypassed, and creating an SSL cert pinned app, what we saw was I forgot the actual name of the process, but it was tunneling, the name of the process, and the destination. And so at that point, we're able to clue in, you know, hey. This is an SSL cert planned app where we need to, create and then send that bypass information over there. With the NPA debug log, we're actually able to see the the application itself, then the rule being pushed down to the client, the policy name, the app name, and if it's being forwarded or blocked and the publishers to which, it's being, connected. K. Within scope it, with page events, one of the things I like to do is take a look at by using a query, for bypass underscore traffic e q u s. And one of the things that this does is it allows me to see what's being bypassed on the local endpoint. In this case, you'll see that the bypass region, is SSL error in complete cert trust chain, and that was, just something that was going on for a Microsoft, update and endpoint as you can see below. With application events, I typically take a look at the activity, the browser, as well as the UA. So with the user agent, this has been really interesting is that we've seen some individuals at times use client switchers or user switcher agents, depending upon which way you phrase it, to potentially bypass some security constraints. And so what I've been doing is I've taken a look at these, user agent strings and I'm actively building out policies, that will stop people from connecting and using these user agents, or the user agent switchers, which you typically is a a Chrome plug in. Right? And this is relevant because not only are we able to stop the, use of that switcher, but more importantly, we can also apply these user agent policies, Real time policies using an HTTP header for a user agent stream to block legacy OSes, such as, you know, iPhone six or, in this case, you know, it would have I think it was Windows Phone eight that they were using. And on network events, we we tend to take a look at the application, the IP protocol, whether it's TCP or UDP, the port and and service traffic stats. Right? With alerts, you know, we'll take a look at alert type, the action, the activity, the application itself, and and the object. Right? So this is an example of alert that we, you know, like, custom sped up or spun up to, you know, see if we were going to encounter any kind of rock star attacks, internally, when taking a look at real time events. And then with DLP incidents, I typically look at the exposure type and the object type. So we're gonna tackle our demo right now. The issue we're seeing is that a user is going to break FTP, and we need to block all possible activities, whether it's related to a web, which is a browse, or if it's related to the cloud app itself. Right? And so we have been experiencing some client update issues on the endpoint. So we're gonna resolve that, and then we're gonna define a custom application, which is relevant because of the, brick FTP CCI information. Then we're gonna steer that custom application, and we're gonna create a cloud access policy as well as a a web access policy. So what we're gonna do is gonna bring up the client. We've been making some changes in the tenant. We've been hoping that the configuration changes have been coming down. Those changes again are with the client configuration itself or steering configuration. Right? As you can see, it's not updating. It's gonna error out. So what I'm doing is I'm going to a command prompt to ping add on -@scope.goscope.com, which is my, the add on for downloading, new updates. Right? I put in a fictitious IP address to show you that, hey. This is this is what it looks like if you can't get connected, to that add on to download these configurations. So as soon as we save this off, then we'll go back to the client itself. K. And once we've got a clear line of sight, this shouldn't be a problem. And this is, indicative of if you've installed any security applications or make DNS changes, that if we can't get a clear line of sight to remote, then, of course, we're gonna have that have that problem and not able able to update or download a configuration. So with this, we've gone to brickftp.com, which redirects to, files.com. We're gonna take a look at the CCI to figure out, okay, what is brickftp.com? Is or doesn't include files.com, or is it just brickftp.com? And you can use this on any of the applications that are out there. Right? If you wanna go in and take a look at the CCI and determine, you know, if it's, discovery only, using a universal connector or an explicit connector. So here, you know, I've seen some issues with related to, searching by domain name. And what we'll do is we're just gonna triple check a few things to make sure that, you know, we're we're covering all our bases, on whether or not that files.com or brickftp.com is what it says it is. So with brickftp.com, I'd like to point out, and I'll pause the video here, that it's discovery only. And so what this means is that we will do our best to discover activities, for this application because it's not using either a, a universal connector, b, an application specific connector, or c, a custom connector. What this means is that depending upon how that application is configured by the provider, we may be able to see uploads, but not downloads. We could see downloads and uploads, but not logins. And then the application provider makes a change, And lo and behold, we can't see any of those. Again, this is kinda the best effort only, and this is relevant as we go through the real time policy definition. Because if an application is discovery only, we cannot create an a real time policy based on that because, again, it's discovery only. There's no guarantee. Right? So as you can see right here, when I try to define brickftp.com, it's not going to work. Alright. So what we need to do is we need to come in and create a custom cloud app, and we're gonna use the connector, the universal connector. And we're gonna add brickFTP and files.com to this to cover all bases. K. Going in and making sure to see if we see any updates. And, of course, we're not. Key here is upper right hand corner. Make sure you apply those changes. I can't tell you the number of times that I personally have been burned by that and forgetting that I haven't applied changes. Case in point is they wanted to make sure this was a live demo. This is about as live as it's gonna get. Right? Or somebody forgets to apply something. Been there, done that. Yeah. So once we've done this, we're gonna update the configuration, right, because we're now pushing down that custom application itself. It usually takes a while on my tenant because I have a lot of, beta stuff going on. So you have to excuse a little delay where we're clicking stuff. And when we now define it, see that it's, in brackets. That's an interesting thing. I'm gonna pause that real quick. Custom applications, right here, as you can see, there's a little four square that's gray and then the name of the application in brackets. Right? And that's to indicate that it is, it is a custom app. So and you'll notice in this that we're not covering the Prowls activity. Just download login variables, log out, and and upload. So when I go through a naming, convention on our policies, I like to say if it's CASB, the next one would be whether it's the application of the category and then, whether or not it's, access or DLP. So threat has a separate category all unto itself, so based on, RBAC groups. So for sake of argument, I will put this at the top again. It just like a firewall, we're gonna process from the top down, and I like to put the most explicit where it's aimed solely at myself. If I'm in a production tenant in this instance, we don't necessarily have to worry too much about it because, it's not a production tenant. K. We're just making sure that we are we are steered. When you're going through these policy changes, I definitely recommend opening up incognito windows and testing. So that way you you're assured that the traffic is truly being, routed through the proxies and we're not having to, know, second guess whether or not it's an existing connection. K. And we're just making sure that the, application is throwing events and we are tunneling traffic. Okay. As you can see right here, recall that I had mentioned that it was a four square, that's gray, and then the application itself is in brackets. And this indicates, of course, it's a it's a custom application. I'm gonna go back in here and collapse that policy, make sure that, you know, we're we're we still have that active, ready to go. I didn't put it at user alert or block. I put it at alert to begin with, and that way we're not impeding, any any user activities. Right? We're just really logging. Then eventually, we'll go to a user alert and then eventually a, a block if you're in a production tenant. And if you don't care about your site getting hammered with alerts, trust me. Doing the user walk right off the bat or use a word will is cause for concern for. I I do care about the socket hammered. Yes. So when we come back into the, application events, you'll notice the orange bullets or dots outside each one of the line items, and scope of events, and that's because, it's it's spawned an alert. Right? So I'm just going through real quick because I love my sock. And I'm like, you know what? What's a few more alerts among friends? Right? It's like, here you go. The point of this, of course, is to determine and make sure that we are you know, we're basically going to log all those events when we see them within the scope of application events. K. They have plenty. Time is relevant. It's 04:40 on my, virtual machine. And then as you can see by the logs, we're seeing a time of events at 04:40. Okay. So what we're now gonna do is we're gonna go back into that policy. We're gonna apply it, and then we're gonna check to see what it looks like after we've blocked. K. You'll notice that browse is not an option in here, and we'll cover that here in a second where we create a, a custom web app or custom web category and apply that to web access policy. So after we create the the URL list, we're gonna apply it to a custom category. And in this instance, since brickftp.com redirected to a files.com, we're gonna add both domain names as well as wildcards at the front just to cover any and all variables that that we may encounter, when we're working with that application or that website in this case. So the one thing I'll say, I'm gonna pause it right now. Be forewarned, don't add it to the URL list down below where it says and not. I've gotten burned a couple of times, one of which I actually opened a support case because I could not, for the life of me, figure out why something wasn't triggering correctly. And they were kind enough to say it's because you added it to the and not, and then it dawned on me. Now we've had some UI changes here recently. And as we push out new, new features, especially when they relate to the UI changes, it's it's definitely important to make sure you're following the logic through the through the policies. Luckily enough, we're we're among I was amongst friends. Otherwise, it might not have, gone over too well. Okay. Now we're gonna go up and we're actually gonna define that web access policy so that we can cover that browse activity. So since we added it as a category, we're gonna go ahead and add that custom here. So you'll see we're covering browse, download, upload, and log in, similar to what we would some of those are similar to the activities we would see in, a cloud app activity. So we're just gonna cover all bases. Sometimes with the, Discovery only and Universal Connector, we might not be able to detect correctly on, some of the activities for whatever reason, using the UC, and we can cover that as well as safeguard, so to speak, when we're digging in, for web. K. We're gonna launch a window, and we're gonna try browse. And there we go. We've been denied. So so the interesting thing about creating these different, the different policies and stacking and where we're taking a look at the application events, sometimes, you know, looking at the refers inside the application event itself is is important so that we can learn or determine, you know, is there another application that we should be blocking prior or another site prior to actually, putting in a web access policy like this. So and with that, ladies and gentlemen, that concludes concludes our demo. How how are we looking on time, miss Paula? Ariana added a little bit of extra time for us. We have a an enormous amount of Q and A. So I just wanted to throw out there that we will do our best to address as many as possible in our time allotted. And if we can't get to them all, we will cover them in our follow-up email. And it will be available in the the community post on the inside Netskope, in the community section right there. Again, we'll continue to try to make sure we we cover everything. And I almost feel like I I totally, like, glossed over a little bit of Stevan introduction here. I wanted to point out that he came to Netskope from a former Netskope customer, and he has been at Netskope for seven years. He is probably the best troubleshooter I have ever met, And just, and he's very transparent as you can tell. We we try to make sure that we share our own, you know, mix ups, goof ups Yeah. And those little things. And and it's just, you know, just, this is why we asked him to do this, just to share some of this just learned knowledge over the years of doing this with, with Netskope. Yeah. It's it's it's been real interesting seeing the, you know, the product line product line develop. And you'll notice that there might have been a hesitation on some of the some of the demo, and that's because I've seen a number of UI changes. Right? You may have seen where I was applying a policy, category change, and the UI might look different from, from y'all. And, you know, the interesting and good thing about working here is we're also working on a lot of alpha beta and pre GA stuff. And so that's another reason why you may have seen my tenant be a little bit slower. And, of course, I'm good at finding problems. And so, you know, that's one of the issues I've, actually found when I was creating this because I flipped on a, you know, they tell you not to do stuff right before a demo. And, of course, I flipped stuff on. Didn't even think about it. So Yes. Indeed. So Open up. So if you could go ahead and, go ahead and stop share, we'll, we'll go ahead and kick up here for a q and a now. Can you provide an explanation on any troubleshooting SOP we can follow to fix Netskope client related issues? Kind of a little bit have gone through there, Stevan. I don't know if you, you got, like, a a workbook in your back pocket? I don't have a workbook. I can I can actually find one, and provide, you know, URLs for our, support, support page? Right? So if you go to docs.netscope.com, we have some really good resources for, you know, what to look for, how to troubleshoot. How I work for, really, a troubleshooting SOP is, you know, can the client, receive updates. Right? You know, up click on it. Does it receive it? Client configuration, yes or no. And then start kinda taking a look at what it looks like from the the NSDEBUG logs, if I really need to get that far down into the weeds, if not if I'm not seeing what I expect inside the scope of application events. So we'll we'll follow-up with, some SOPs and, some resources that you can put in your back pocket and and refer to later. Nice. Can we see a way to troubleshoot logs and events alerts scope it? It is sometimes difficult to know where to start troubleshooting events that have been allowed or sometimes blocked if the end user is not able to send the technical details of what occurred. Yeah. I think we covered a little bit of that in here, Stevan. I don't know if you wanna give a couple more details on, specifics. I know I always, I always isolate it, you know, when I'm doing the scope of searches to the user. And if they've described the activity, I'm that's what I'm gonna try to, limit it down to make the the focus set of alerts. But, I mean, Stevan does this so much more than I do. Yeah. The, the one thing, you know, I've I've noticed that if the end user can't provide you the details, is to get a screen recording. And this will help out immensely when you're working with our support team, because they're inevitably gonna ask for, you know, can you provide a recording or a screenshot of the the error message? And if the end user can't or sometimes won't send you client logs, you have the ability to collect those through the, through the actual tenant itself. Alright. What's the best way to read client logs? I'm a Windows guy, so I'm I'm in, net Notepad plus plus using the, you know, search features and whatnot in there. So, Stevan? The smart Alec response to would have been really carefully. That works? Yeah. Very carefully. Very carefully indeed. Save the Windows, save the Windows logs off. I'm in Notepad plus plus. I love Windows Notepad plus plus. However, you'll get that annoying, this file has changed, do you wanna reload its contents? With Mac, Macs are a little bit more for or, forgiving. You can go with advanced debug, open up the log file, and while events are happening, they'll stream through and and not, not alert you. So I'm sure there's a setting you can flip in Notepad plus plus or another, log utility, but I just I I choose to save them off. It's also a good, way to, stop time because you may be encountering an issue at a certain point in time, and then, of course, things change, especially if it's a a, a machine that has a lot of traffic. So while you have this issue here, stop time, capture it, and then you can do that same thing again if something's changed. Right? Maybe you make a policy change, or client configuration change, and then you stop seeing, you you stop seeing those alerts. So that way you can do that that comparison. Yep. Could we get a better understanding of what the different advanced debugging options do? It would be great to have some documentation including examples and suggestions on when to use different levels. The only guidance I have found is that it should be left at info by default. I think we covered this one pretty much in the in the webinar. Info is ordinarily everything you need, to do some troubleshooting and really only only turn it on, you know, for the the debug if asked for from support. As a new Netskope admin user, what are the most tactile resources to use for solidifying the knowledge and gaining confidence tackling every and any issue that arises? You know, for me, when I first started with it, I definitely had to learn, you know, reading those, those client logs were were crucial in identifying issues, and, you know, what what the client might be seeing as far as traffic, what it could be interfering with, as far as app local applications, and just getting very familiar with what the policies are capable of. So, Stevan, I can pass this over to you if you wanna get take a shot of this one too. Yeah. Definitely. Dogs.nesco.com, and our community resources. Those are those are great, great resources. When I first started, we didn't have, either one of those. And so I would always have to go in and, you know, kinda, you know, reach out to Slack channels or dig into engineering documents. And, thankfully, they've been able to, you know, bubble this up to the surface and expose a good amount of information so that it can give you a really, really solid platform on, gaining knowledge and experience. And my recommendation also is to not be afraid to break stuff, especially with policies. Right? Make sure that you're putting the most specific policy as possible, only affecting you at the top on top of the list. Right? And then going through that troubleshooting stance, and then, you know, making those policy changes for the rest of the organization. So Yep. Yep. How can we get accurate reporting notifications of clients tunnel down due to error? I'll let you Stevan. So tunnel down due to error, you can go into the Netskope settings, security cloud platform devices, and take a look at device status in there. If you have advanced analytics, we actually have a report, that you can run on client statuses. And I believe advanced reporting has a trimmed down version of that same, Netskope advanced analytics report. Nice. How can I get a client enrolled in multiple Netskope tenants? That one just doesn't work. Sorry. It it doesn't. The only the only response to that would be uninstall and install a new client. Yep. How do I troubleshoot my Internet being slow? Speed test says I have 200 meg down when I have a gig at home. So with the Internet slow speed, my recommendation is to, take packet captures, you know, screenshot and or video, and then note what pop you're connected to, which you can also get from a screenshot of your, configuration, and then open up a potential, ticket with, the Netskope client team. So I've seen oh, go ahead. I was just gonna say I I would also say that, one of the tools that I've been leveraging a lot lately when I have a a user says, hey. My Internet's slow, is we have PDM enabled in our tenant, and we're able to, you know, target target and go see what's actually happening with their traffic. And one of them, they were like, man, everything's just terrible. They live on the West Coast, and all their AWS stuff was routing through the East Coast. And it's like, what what is happening? But PDM was able to, you know, give us the view of what's going on. And so we were able to, you know, figure out why their experience was so poor and, take some steps to correct it. So or if there's nothing we can do, at least we have the visibility to say it's out of our hands. We can't fix this. That's a good call out on, on PDM. Sometimes I forget the tools that we have at hand. So that's a good call out on that one. Hey. Nicholas, it's One of one of alright. One of our partners are, man. Stevan, nice to see you again. There's a way we can see in client logs the cause of a network latency. We have PDM. However, wanna know if we can also see the cause or sign of a network latency in client logs because it's important to capture the issue near real time when it comes to latency issues. So I would take a look at that in in the nsdebuglog.log and the old .log, to determine your round trip times, because you're gonna be able to take a look at your, your GSOB endpoints throughout the world, and, of course, it's picking the one with the lowest round trip time. We've actually seen an issue where one of our executives went on vacation and, they said, well, hey. I'm, you know, I'm being routed. I think it was Chicago, which is IAD or IAD, and then they were being routed down to over to Philadelphia, and I think it was down to Miami. And then after looking in the NSDEBUG log file, it was saying, okay. Well, the round trip times, here's the reason why because, you know, they were bounced all around, and those round trip times were being picked based on best, best connection speed or best RTT. So that's where I would take a look in the NSDEBUG log file. So How to read logs and which type of logs to look at? So how to read the logs. So we think we already covered that. And, definitely, the two logs, that Stevan has already called out, the NPA debug and NS debug, are the ones you're primarily gonna rely on. Or the smart aleck response very carefully with your eyes. Very carefully. How do you read them? How do you describe the icon that people should look for? Is is the configuration the same as manually updating the app? Yeah. The the icons, the little, colored Netskope, Netskope symbol, for the for the client. And when you bring up the configuration, you have a choice of manually updating it. Or I believe it's every fifteen minutes, it checks and phones in and pulls down a new configuration. And then as far as, that's for the client config and the steering config. As far as the application itself goes, though, every four hours at four hours of inactivity, it reaches out to the tenant, says, hey. Do you have a new version? If it does, it'll pull it down and install it at that point. So Cool. How to rectify big file malware scanning error? Is there a place for me to check SSL related errors? I can't seem to find it and Netskope. Yeah. With Netskope, that there's a, alert policy type malware. You also may be able to see within the page events if it's an s o SSL, SSL error. As I mentioned earlier, you saw in the example where it was bypass traffic, and it was because of, I think it was SNI with Microsoft. We're I forgot the actual specific here. So but you should be able to pull some of this out of, Netskope. What I can do is, make it a make it a line item, and I'll, you know, follow-up to provide resources, for this question after we're done here. Alrighty. How to troubleshoot issues with uninstallation process for older Netskope clients? Netskope client unable to be removed. Yes. Yes. I was, recently encountered this one and was humbled because in the past, I would go through registry editor. And I don't know if anybody's messed in the Windows registry before, but it was it was painful. But somebody said, hey. Have you tried renaming this folder? And that folder was in, I think it's c windows. I'll have to pull out that pull out the folder, but I think it was actually in a in a users users folder. After renaming it, I was able to uninstall the uninstall the old client. So I'll follow-up with resources on that after we're done. Yes. When Windows has always got extra tricks you can do, man. So can you explain the difference between the outer PCAP log versus a Wireshark log? Sometimes support asks explicitly Wireshark, and I'm curious what the difference is. Yeah. That's that's interesting. I've never heard the them differentiate between the two. They may be talking about, the outer and the inner. Right? Because of those well, actually, there's three. There's an outer and inner, and there's also an NPA, PCAP. Right? So I'm assuming that the Wireshark log that they're referring to is the the inner inner PCAP. And the reason behind that is, that's what, you know, the browser or the user agents are reaching in and you're seeing on the inside of that network. The outer would be what the client is sending to the actual tenant itself. So so they're probably talking about an inner inner log, inner PCAP. Scope, it will log all bypass traffic, including traffic that is bypassing steering? You if you go into the steering configuration, there's gonna be, a setting up at the top, and it says, bypass traffic, or bypass traffic log. So you will actually have to edit the steering configuration to log a bypass traffic so that when you go into page events, you can do a query that's bypass underscore traffic e q yes to see what traffic's being, being actually bypassed within that. So a majority of the bypass traffic will be logged. I, I like how you actually have that query memorized. Yeah. And that's that's my favorite query. As soon as they introduced that into the logic and page events, it made it so much nicer, so much more pleasant. Alright. How would you go about troubleshooting a Microsoft Teams call performance issue to confirm if Netskope proxy is the root cause or is something else? For me, that's when I immediately go into, PDIM for. Find the user, target the, you know, the app Teams application, and see if I can find out where the performance issues are are going. Stevan, I'll let you take a shot at that one too. No. That's that's actually a a great example of PDIM. So So and, again, we we don't know what everybody's licensed for. We just know what we use. So we're not we're not trying to upsell anybody. We're just, hey. These are the tools that we utilize day in and day out. So, But but Alright. You want but if you want, PDM, tell them Stevan and Rob sent you. I I wish we got commissions. So, anyways, if you cannot find any object by the DLP incident ID, what is the best way to find it in Netskope? Searching by object object name, or, you know, person the offending person. That's how I would go about looking for a DLP incident. Right? We've actually had, issues or I've seen issues where, sometimes that that DLP incident ID might be bundled in something else or is not readily available, but you can always take a look at the object object itself. Alright. Is there a solution to force Netskope to switch multiple devices greater than a hundred to a specific POP in case of issues with one POP? It is my understanding that, our GSLB is supposed to detect those issues and reroute you to a new POP, very quickly. And, if I'm if I'm wrong or mistaken here, or if I'm missing some information, Stevan, please, fill fill me in. No. That's that's it. It that's it in a nutshell. We don't have an ability to force switch multiple clients. So Yeah. When will alert action for browse activities be available? Browser. Yeah. I'm trying having a hard time wrapping my mind around that. Right? Because, I just demoed what it would look like for, alert on a browse activity, with that custom app that I had earlier. Alright. So I think I think we've actually covered that one then since you showed it in the demo. There may there may be, other mitigating circumstances. What I'll do is I'll dig because I like to experiment. I'll dig around later today and and see if I can trigger trigger an alert for other browse activities. Where can we we view the inner outer packet capture logs? Which logs show the RTT and DC connections? So to view the inner and outer packet capture logs, I would basically go to advanced debugging, reveal logs, and then it would open up the folders in which those log files are gonna, belong. Or you can always save, the log files off to a ZIP and then open that ZIP and then examine those. As far as the RTT and DC connections, I would actually look in the NS debug log, and NS debug old log files. Alright. Can you please explain how to troubleshoot scenarios where websites remain inaccessible and there is no Netskope error message? So Let's talk about how to read and interpret the Netskope logs. Yeah. Okay. So let me let me think through that scenario. Right? Remains inaccessible. So I would take that URL, and then I would go in the scope it, and I would do a URL check to see what category it might belong to. Then you can jump into, application events and do a query on URL like and then the name of that URL and see if you're, spawning events, for alerting. Sometimes those alerts don't necessarily become readily available. And so at that point, you know, you can start digging in and taking a look at the page events for that host. And depending upon what alert alerts you do see, will depend on, you know, the next course of actions. Right? You can also look in the NSDEBUG log. See, I love that NSdebug log file. You can actually dig in and see that host it's going to and what process it's, is sending and communicating with that remote destination. And with that, we are out of time. If we can take a look at our poll and see what, we're bringing to you next time around. Alright. We are gonna be looking at, Netskope endpoint SD WAN. You will be entertained by Stevan once again, because this has been one of his projects he's been working here greatly. So good good interactions. Thank you for all the questions. Sorry we didn't have enough time to get through all of them, but we will, have a summary response for everything to the community post in short order. Thank you again, and, have a fantastic rest of your day.