Duplicates in dictionary python

Источник

How to find repeats or duplicates in nested dictionary?

because this dictionary contains duplicates. I was able to do this quite easily with a regular dictionary, and I thought this would work well with this case too:

dictionary = >> rev_dictionary = <> for key, value in dictionary.items(): rev_dictionary.setdefault(value, set()).add(key) print(rev_dictionary) for key,values in dictionary.items(): if len(values) > 1: values = True else: values = False 

TypeError: unhashable type: 'dict' 

How can I get this working? Thanks for the help! Note: I'd prefer a solution without using libraries if possible

Define "duplicates." In my opionion, even if I am given , "2": > , these are still two distinct objects, regardless of the fact that they share some commonalities.

5 Answers 5

I am assuming you are defining duplicates by value and not by keys. In that case, you can flatten the nested dict using (mentioned here)

def flatten(d): out = <> for key, val in d.items(): if isinstance(val, dict): val = [val] if isinstance(val, list): for subdict in val: deeper = flatten(subdict).items() out.update() else: outDuplicates in dictionary python = val return out 

and then check for the condition

v = flatten(d).values() len(set(v))!=len(v) 

I wrote a simple solution:

dictionary = >> def get_dups(a, values=None): if values is None: values = [] if (a in values): return True values.append(a) if type(a) == dict: for i in a.values(): if (get_dups(i, values=values)): return True return False print(get_dups(dictionary)) 

How it works

We start by saving every value in a list, which we will pass into the function. Each run we check whether our current value is in that list, and return True once there is a duplicate.

if (a in values): return True 

Next we just loop through the values and run get_dups on them if the current index is also a dictionary.

You can recursively add item values of sub-dicts to a set, and if any item value is already "seen" in the set, raise an exception so that the wrapper can return True to indicate that a dupe is found:

def has_dupes(d): def values(d): seen = set() for k, v in d.items(): if isinstance(v, dict): s = values(v) if seen & s: raise RuntimeError() seen.update(s) else: if v in seen: raise RuntimeError() seen.add(v) return seen try: values(d) except RuntimeError: return True return False 

so that given your sample input, has_dupes(dictionary) would return: True

I think all you need is to flatten the dict before passing to your duplication-detection pipeline:

import pandas as pd def flatten_dict(d): df = pd.io.json.json_normalize(d, sep='_') return df.to_dict(orient='records')[0] dictionary = >> dictionary = flatten_dict(dictionary) print('flattend') print(dictionary) rev_dictionary = <> for key, value in dictionary.items(): rev_dictionary.setdefault(value, set()).add(key) print('reversed') print(rev_dictionary) is_duplicate = False for key, values in rev_dictionary.items(): if len(values) > 1: is_duplicate = True break print('is duplicate?', is_duplicate) 

flattend reversed , None: , 5: > is duplicate? True 

Источник

5 ways to Remove duplicates list of Python dictionaries

In this post, we will understand the different 5 ways to Remove duplicates list of Python dictionaries with examples. Here we are using simple for loop, list comprehension,frozenset(), and many more techniques.

1. For loop to Remove duplicates list of Python dictionaries

The simple and easy method comes to our mind to loop over lists of dictionaries and compares the value for a duplicate to remove the duplicate value.

In this code example, we are looping over the lists of dictionaries and removing the duplicate dictionary, and appending it in the new dictionary.

Let us understand with code examples and build our understanding from it.

Program Example

#Program to remove duplicates from a list of dictionaries Langlist = [, , <"C++" : 3>, , ,] #removing the duplicate entry unique_dict = [k for j, k in enumerate(Langlist) if k not in Langlist[j + 1:]] print('after Removing duplicate from list of dictionary =\n',unique_dict)

Output

after Removing duplicate from list of dictionary = [, <'C++': 3>, , ]

3. Remove duplicate lists of dictionary by convert list to tuple

In this code example, we are converting lists of dictionaries into a list of tuples, and further using the set Comprehension to remove the duplicate from the lists of dictionaries.

Let us understand with code examples and build our understanding.

Program Example

#Program to remove duplicates from a list of dictionaries Langlist = [, , <"C++" : 3>, , ,] #removing the duplicate entry unique_dict = [dict(tuple) for tuple in ] print('after Removing duplicate from list of dictionary =\n',unique_dict)

Output

after Removing duplicate from list f dictionary = [, , , <'C++': 3>]

4. Using package iteration_utilities.unique_everseen

In this code example, we are using the third-party package to remove duplicates from lists of dictionaries. We are importing the package iteration_utilities import unique_everseen and using its unique_everseen() function to remove duplicates.

Let us understand with code example below.

Program Example

#Program to remove duplicates from a list of dictionaries from iteration_utilities import unique_everseen Langlist = [, , <"C++" : 3>, , ,] #removing the duplicate entry unique_dict=list(unique_everseen(Langlist)) print('after Removing duplicate from list of dictionary =\n',unique_dict)

Output

after Removing duplicate from list of dictionary = [, , <'C++': 3>, ]

5. frozenset() to Remove duplicates list of dictionaries

frozenset() is an inbuilt function of python it takes iterable as an argument and makes it immutable(unchangeable) or it freezes the iterable object and makes it unchangeable. It returns an immutable frozenset() object initialize with the elements from the iterable.

In this code snippet, we are using the frozenset() to remove duplicates from lists of dictionaries.

Let us understand with the code example below.

Program Example

#Program to remove duplicates from a list of dictionaries Langlist = [, , <"C++" : 3>, , ,] #removing the duplicate entry unique_dict = .values() print("after Removing duplicate from list of dictionary ="+str(unique_dict))

Output

after Removing duplicate from list of dictionary = dict_values([, , <'C++': 3>, ])

Conclusion

In this post, we understood the different ways to remove duplicates from the list of dictionaries. We hope you will find these ways useful and these will help you to solve your problem.

These techniques will be really helpful when you are working on the Data analysis work and they help you clean your data set samples.

Источник